Imaging apparatus and method, operation apparatus and method, and program and recording medium

ABSTRACT

A luminance variation (dI) pertaining to each pixel is calculated ( 21 ) using a plurality of captured images obtained by image capturing under different illumination conditions, a texture variation (dF) pertaining to each pixel is calculated 
     ( 22 ) using a plurality of captured images obtained by image capturing at different time points, and a subject region is extracted ( 23 ) based on the luminance variation (dI) and the texture variation (dF). A variation in the texture feature (F) pertaining to each pixel between the images is calculated as the texture variation (dF). The subject can be extracted with a high accuracy even when there are changes in the ambient light or the background.

The present invention relates to an imaging apparatus and method forextracting a subject region from a captured image. In particular, thepresent invention relates to a technique for extracting a subject regionusing differences between two or more captured images obtained by imagecapturing under different illumination conditions. The present inventionalso relates to an operation apparatus and method for performingoperation on a device using data obtained by the above-mentioned imagingapparatus. The present invention also relates to a program for causing acomputer to execute processes in the above-mentioned imaging apparatusor method, or the above-mentioned operation apparatus or method, and acomputer-readable recording medium in which the above-mentioned programis stored.

BACKGROUND ART

When a person in the vicinity of a camera or a part (a hand, forexample) of such a person is imaged as a subject, for the purpose ofmonitoring the state or action of the person, it is effective to removethe background and extract the region which the subject occupies, andanalyze the extracted region in detail. Patent reference 1 discloses adisplay imaging apparatus which periodically turns on and off theilluminating light for irradiating the subject, at a short interval, andobtains the captured image during the on-period, and the captured imageduring the off-period, and uses the difference between them to extractthe region of the subject in the vicinity the camera.

A problem associated with this method is that when the backgroundchanges or the subject moves, a region of false signals may appear inthe difference image. To cope with this problem, Patent reference 1discloses generating an interpolated image in which the movement of thesubject or the background has been compensated for, and generating adifference image, from which the regions of false signals due tomovement of the subject have been removed. Also, Patent reference 2discloses an image processing apparatus which detects the movementamount of the subject, and uses the detected movement amount for theidentification of the subject region.

PRIOR ART REFERENCES Patent References

-   Patent reference 1: Japanese Patent No. 4915367-   Patent reference 2: Japanese Patent No. 4842374

Non-Patent References

-   Non-patent reference 1: Marko Heikkila, et al., “Description of    Interest Regions with Local Binary Patterns”, 30 Jun. 2008

Non-patent reference 1 will be mentioned later.

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

However, the technique for extracting a subject region using adifference image has a problem in that in a space in which the ambientlight or the background changes rapidly, such as a space inside of amoving vehicle, or out of doors, the subject cannot be correctlyextracted due to the effects of the false signals. In Patent reference1, it is necessary that the same subject and the same background objectsare present in the plurality of images used for the generation of theinterpolated image, for the purpose of compensating for the movement ofthe subject or the change in the background. However, in a movingvehicle, or out of doors, the change in the ambient light or the changein the background is fast, and the same subject or the same backgroundis not necessarily present in the images, and in such a case, theeffects of the false signals cannot be eliminated.

In a case in which the subject region is identified by detecting themovement amount of the subject, the luminance variation calculated bytaking the difference: is susceptible to the ambient light, and, it isdifficult to determine whether the change in the luminance is due tofalse signals, inside of a vehicle, or out of doors. Also, similar falsesignals are generated due to changes in the background, and the regionin which the false signals are generated may be erroneously taken as aregion of the subject.

The present invention is to solve the problems described above, and itsobject is to enable extraction of the subject with a high accuracy, evenwhen there is a change in the ambient light or background.

Means for Solving the Problems

An imaging apparatus according to the present invention comprises:

an imaging/irradiating control unit for generating an illuminationcondition control signal for controlling an illumination condition, andan imaging condition control signal for controlling an imagingcondition;

an irradiating unit for illuminating a subject with a plurality ofmutually different illumination conditions based on the illuminationcondition control signal;

an imaging unit for performing image capturing of the subject with animaging condition controlled by the imaging condition control signal togenerate captured images;

a luminance variation calculation unit using a plurality of capturedimages obtained by the image capturing under the different illuminationconditions by said imaging unit for calculating a luminance variationpertaining to each pixel between the plurality of captured images;

a texture variation calculation unit using a plurality of capturedimages obtained by the image capturing at different time points by saidimaging unit for calculating a texture variation pertaining to eachpixel between the plurality of captured images; and

a subject extraction unit for extracting a subject region based on theluminance variation and the texture variation.

Effects of the Invention

According to the present invention, the subject can be extracted with ahigh accuracy even when there is a change in the ambient light or thebackground.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the configuration of an imagingapparatus of a first embodiment of the present invention.

FIG. 2 is a block diagram showing the configuration of the irradiatingunit in FIG. 1.

FIG. 3 is a block diagram showing the configuration of the luminancevariation calculation unit in FIG. 1.

FIG. 4 is a block diagram showing the configuration of the texturevariation calculation unit in FIG. 1.

FIG. 5 is a diagram showing an arrangement of pixels used for thecalculation of a CSLBP feature.

FIGS. 6(a) to 6(h) are diagrams showing the results of calculation ofthe CSLBP features on images captured under different illuminationconditions, and luminance histograms of the images.

FIG. 7 is a block diagram showing the configuration of the texturefeature calculation unit in FIG. 4.

FIG. 8 is a diagram showing a method of dividing the feature extractionregion into cells.

FIG. 9 is a block diagram showing the configuration of the subjectextraction unit in FIG. 1.

FIG. 10 is a diagram showing a relationship between the change in theillumination condition, the change in the background, and the movementof the subject, and the luminance variation and the texture variation.

FIGS. 11(a) to 11(e) are diagrams showing an example of extraction of asubject region by the imaging apparatus of the first embodiment.

FIGS. 12(a) to 12(e) are diagrams showing an example of extraction of asubject region by the imaging apparatus of the first embodiment.

FIG. 13 is a flowchart showing an exemplary processing procedure in theimaging apparatus of the first embodiment.

FIG. 14 is a flowchart showing an exemplary procedure of the subjectextraction step in FIG. 13.

FIG. 15 is a flowchart showing another exemplary procedure of thesubject extraction step in FIG. 13.

FIG. 16 is a block diagram showing the configuration of a variation ofthe texture variation calculation unit of the first embodiment.

FIG. 17 is a flowchart showing a processing step in a variation of theimaging apparatus of the first embodiment.

FIG. 18 is a flowchart showing a processing step in a variation of theimaging apparatus of the first embodiment.

FIG. 19 is a diagram showing an arrangement of pixels used for thecalculation of a texture feature in a variation of the first embodiment.

FIG. 20 is a flowchart showing another exemplary processing procedure ina variation of the imaging apparatus of the first embodiment.

FIG. 21 is a block diagram showing the configuration of an imagingapparatus of a second embodiment of the present invention.

FIG. 22 is a block diagram showing the configuration of the target valuecalculation unit in FIG. 21.

FIG. 23 is a flowchart showing a processing procedure in the imagingapparatus of the second embodiment.

FIG. 24 is a flowchart showing a processing procedure in the imagingapparatus of the second embodiment.

FIG. 25 is a block diagram showing the configuration of an imagingapparatus of a third embodiment of the present invention.

FIG. 26 is a block diagram showing the configuration of the luminancevariation calculation unit in FIG. 25.

FIG. 27 is a block diagram showing the configuration of the texturevariation calculation unit in FIG. 25.

FIG. 28 is a flowchart showing a processing procedure in the imagingapparatus of the third embodiment.

FIG. 29 is a flowchart showing a processing procedure in the imagingapparatus of the third embodiment.

FIG. 30 is a block diagram showing the configuration of an operationapparatus of a fourth embodiment of the present invention.

FIG. 31 is a block diagram showing the configuration of the texturevariation calculation unit in FIG. 30.

FIG. 32 is a block diagram showing the configuration of the subjectrecognition unit in FIG. 30.

FIG. 33 is a diagram showing an exemplary correspondence between thegesture types recognized by the subject recognition unit in FIG. 30, andthe contents of the operation generated by the operation determinationunit.

FIG. 34 is a flowchart showing a processing procedure in the operationapparatus of the fourth embodiment.

FIG. 35 is a block diagram showing a computer used for implementing theimaging apparatus in the first embodiment by software, together with anirradiating unit and an imaging unit.

MODES FOR CARRYING OUT THE INVENTION First Embodiment

FIG. 1 is a block diagram showing the configuration of an imagingapparatus according to a first embodiment of the present invention. Theillustrated imaging apparatus includes an imaging/irradiating controlunit 11, an irradiating unit 12, an imaging unit 13, a luminancevariation calculation unit 21, a texture variation calculation unit 22,a subject extraction unit 23, and an image memory 14.

The imaging apparatus of the present embodiment takes a picture of, forexample, a person, as a subject. Specifically, the entire body of aperson, or a part, such as an upper body, a face, or a hand is separatedas a subject, from the background.

The imaging/irradiating control unit 11 outputs a control signal(illumination condition control signal) C11 a for controlling theillumination condition to the irradiating unit 12, and also generates acontrol signal (imaging condition control signal) C11 b for controllingthe imaging condition and outputs it to the imaging unit 13.

The illumination condition controlled by the control signal C11 aincludes at least one of the irradiation distribution of theilluminating light, the light emitting intensity of the illuminatinglight, the light emitting timing of the illuminating light, and thelight emitting, period of the illuminating light.

The imaging condition controlled by the control signal C11 b includes atleast one of the exposure timing, the exposure time, the frame rate, theaperture, and the gain.

The imaging unit 13 performs image capturing frame by frame. Switchingof the illumination condition by the irradiating unit 12 is performed insynchronism with the imaging by the imaging unit 13.

In the present embodiment, the control signal C11 a supplied from theimaging/irradiating control unit 11 alternately designates, frame byframe, a different one of the illumination conditions A and B, and theirradiating unit 12 performs the illumination with an illuminationcondition A and the illumination with an illumination condition Balternately frame by frame based on the control signal C11 a. Performingillumination with a certain condition is also termed as generating suchan illumination condition.

As shown in FIG. 2, the irradiating unit 12 includes a plurality of LEDs121 and a light emission control unit 122.

As the LEDs 121, near-infrared LEDs, or white-light LEDs are used.

When the subject is the upper body of the person, it is desirable thatnear-infrared LEDs are used as a light source since human eyes havelittle sensitivity to the infrared rays.

For changing the light emitting intensity of the illumination by theirradiation unit 12, the number of the LEDs which are turned on to emitlight, among the plurality of the LEDs 121, may be changed, or the lightemitting intensity of each LED may be changed, or both methods may beused in combination.

For changing the light emitting intensity of each LED, the magnitude ofthe drive current flowing into the LED may be changed, or the duty ratioin the PWM (Pulse Width Moduration) control may be changed.

For changing the irradiation distribution of the illuminating light bythe irradiating unit 12, the LEDs which are turned on to emit light maybe altered, or the light emitting intensity of each LED may be changedaccording to its position (the position within the group of theplurality of LEDs 121).

In this following description, it is assumed that the switching of theillumination condition is made by controlling the duty ratio in the PWMcontrol thereby to switch the light emitting intensity.

The light emitting intensity in the illumination condition A is denotedby ϕA, and the light emitting intensity in the illumination condition Bis denoted by ϕB. It is assumed that the light emitting intensity ϕA andthe light emitting intensity ϕB are respectively fixed values, and havethe following relationship:ϕA>ϕB

The imaging unit 13 performs image capturing of a subject based on thecontrol signal C11 b supplied from the imaging/irradiating control unit11, and under the two illumination conditions generated by theirradiating unit 12, and causes the images G obtained by the imagecapturing to be stored in the image memory 14.

As the imaging unit 13, an imaging element such as a CMOS sensor, a CCDsensor is used.

The imaging unit 13 performs image capturing based on the control signalC11 b, at a rate of, e.g., 30 frames per second. The output images Gare, for example, gray-scale images (black-and-white images), or RGBimages.

In the following description, it is assumed that the imaging unit 13outputs images at a rate of 30 frames per second. The output images are,for example, gray-scale images of eight-bit gradation.

The resolution of the output images is, for example, according to theVGA standard, and the width W of the image is 640 pixels, and the heightH is 480 pixels.

The irradiating unit 12 and the imaging unit 13 are so disposed that thelight (illuminating light) irradiated from the irradiating unit 12 isreflected by the subject and reflected light enters into the imagingunit 13.

The imaging unit 13 performs image capturing alternately under theillumination condition A and the illumination condition B which aregenerated alternately by the irradiating unit 12, and alternatelyoutputs the image (image of the illumination condition A) Ga obtained bythe image capturing under the illumination condition A, and the image(image of the illumination condition B) Gb obtained by the imagecapturing under the illumination condition B. As a result, the images Gaof the illumination condition A and the images Gb of the illuminationcondition B are output at a rate of 15 frames per second.

The images G (Ga, Gb) output from the imaging unit 13 are successivelystored in the image memory 14.

It is assumed that each of the imaging condition for the image capturingunder the illumination condition A, and the imaging condition for theimage capturing under the illumination condition B is fixed.

The luminance variation calculation unit 21 reads, from the image memory14, two images Ga, Gb which the imaging unit 13 obtained by capturingone after the other, under different illumination conditions, comparesthe luminance values I for each pixel, calculates and outputs aluminance variation dI between the images. The output luminancevariation dI is supplied to the subject extraction unit 23.

As shown in FIG. 3, the luminance variation calculation unit 21 includesa luminance feature quantity calculation unit 211, a luminance featurequantity memory 212, an illumination condition determination unit 213, adifference calculation unit 215, and a luminance variation memory 216.

The luminance feature quantity calculation unit 211 calculates theluminance feature quantity Im of the image of each frame read from theimage memory 14. The luminance feature quantity Im is, for example, aluminance mean value, a luminance median value, or a luminance modevalue. The luminance feature quantity Im calculated by the luminancefeature quantity calculation unit 211 is stored in the luminance featurequantity memory 212, and read by the illumination conditiondetermination unit 213, one frame period later, as the luminance featurequantity Im of the image of the preceding frame.

The illumination condition determination unit 213 reads the luminancefeature quantity Im pertaining to the latest image (image of the latestframe) calculated by the luminance feature quantity calculation unit211, and the luminance feature quantity Im pertaining to the image oneframe before, from the luminance feature quantity memory 212. Theseimages are two images having image capturing time points adjacent toeach other, i.e., images of two frames one after the other. Theillumination condition determination unit 213 compares the luminancefeature quantities Im pertaining to the images of the two frames, anddetermines which of them is an image Ga of the illumination condition A,and which is an image Gb of the illumination condition B. The result ofthis determination, CNa, is supplied to the difference calculation unit215.

For instance, it is determined that the image with the larger luminancefeature quantity Im is an image Ga of the illumination condition A, andthe image with the smaller luminance feature quantity Im is an image Gbof the illumination condition B.

The difference calculation unit 215 determines a difference in theluminance I for each pixel, between the image Ga of the illuminationcondition A and the image Gb of the illumination condition B, andoutputs the difference as the luminance variation dI. That is, thedifference calculation unit 215 calculates, for instance, the luminancedifference dI(x,y) between the two images, from the luminance valuesI(x,y) for each pixel in the images.

Here, x and y denote coordinates in the images, and are related asfollows:x∈(0,1, . . . ,W−1),y∈(0,1, . . . ,H−1)

The luminance difference dI(x,y) is calculated by subtracting, from theluminance value of each pixel in the image Ga of the illuminationcondition A, the luminance value of the pixel at the same position inthe image Gb of the illumination condition B. Which of the two imagesread from the image memory 14 is the image Ga of the illuminationcondition A is decided based on the determination result CNa output fromthe illumination condition determination unit 213.

If the result of the subtraction is a negative value, the luminancedifference dI(x,y) is treated as being zero.

The luminance difference dI(x,y) calculated in this way is called aluminance variation.

The luminance difference dI(x,y) is stored as the luminance variation inthe luminance variation memory 216. The stored luminance differencedI(x,y) is later supplied to the subject extraction unit 23.

A set of the luminance differences dI(x,y) for all the pixels, i.e., theluminance differences dI(x,y) arranged at the same positions as therespective pixels, is generated as a luminance difference image(luminance variation image).

The images output from the imaging unit 13 contain noise due to darkcurrent or charge reading noise in the imaging elements. For example,the luminance variation calculation unit 21 may perform smoothingfiltering, using a smoothing filter, not shown, on the images suppliedfrom the imaging unit 13, and may thereafter determine the difference atthe difference calculation unit 215. Alternatively, the luminancevariation calculation unit 21 may remove the noise by performingsmoothing filtering by means of a smoothing filter, not shown, on theluminance difference image calculated by the difference calculation unit215, and output the result of the filtering as the luminance variation.As the smoothing filter, a Gaussian filter, a median filter or the likemay be used.

By performing the above-mentioned smoothing filtering before calculatingthe luminance differences, noise due to the camera and the like can beremoved. When the above-mentioned smoothing filtering is performed aftercalculating the luminance differences, the regions where the differencesare small (that is noise) can be removed. Also, the above-mentionedsmoothing filtering may be performed both before and after the luminancedifference calculation.

By performing a threshold processing on the luminance difference image,a subject region can be extracted. However, if the subject regionextraction is performed based only on the luminance difference image,the part where a change in the background, or movement of the subjecthas taken place may be erroneously determined to be a subject region.The part which is erroneously determined to be a subject region iscalled a region of false signals.

As will be described below, according to the present invention, atexture variation is also used to distinguish between the region offalse signals and the region of the subject, enabling extraction of thesubject region only.

The texture variation calculation unit 22 successively reads a pluralityof images stored in the image memory 14, determines a texture feature Ffor each of the pixels constituting each image, calculates a texturevariation dF between two images obtained by image capturing at twodifferent time points, and supplies the calculated texture variation dFto the subject extraction unit 23.

The above-mentioned “two different time points” are, for example, frameperiods which occur one after the other. Accordingly, the imagescaptured at two different time points are, in the present embodiment,images Ga, Gb under two different illumination conditions. However, inthe calculation of the texture variation dF, it is not essential thatthe illumination conditions are different, but what is essential is thatthe time points are different. Accordingly, to emphasize this aspect,the expression “two different time points” are sometimes used.

As shown in FIG. 4, the texture variation calculation unit 22 includes aCSLBP feature calculation unit 221, a CSLBP feature memory 222, atexture feature calculation unit 223, a feature variation processingunit 225, a texture feature memory 224, and a texture variation memory226.

The CSLBP feature calculation unit 221 reads, from the image memory 14,an image captured at each time point (frame period), calculates a CSLBPfeature B for each pixel in the image having been read, and causes thecalculated CSLBP feature B to be stored in the CSLBP feature memory 222.

The texture feature calculation unit 223 reads the CSLBP feature Bpertaining to each pixel in each image stored in the CSLBP featurememory 222, calculates the texture feature F pertaining to each pixel inthe image, and causes it to be stored in the texture feature memory 224.

The feature variation processing unit 225 reads the texture features Fpertaining to the same pixel (pixels at the same positions) in the twoimages from the texture feature memory 224, calculates the variation(texture variation) dF in the texture feature F between the images, andcauses the calculated texture variation dF to be stored in the texturevariation memory 226. The stored texture variation dF is later suppliedto the subject extraction unit 23.

The texture feature F calculated by the texture feature calculation unit223 is a feature which relates to the appearance of the subject or thebackground, and represents the patterns, the unevenness, or thereflectivity of the surface, which is little dependent on theillumination condition.

The texture feature F is, for example, represented by a feature vectornumerically expressing the relationship, with respect to each pixel(pixel of interest) in the image, of the luminance values of the pixelsat specific positions in a region (feature extraction region) centeredon the pixel of interest.

The feature vector which can be used as one representing the texturefeature F is desirably a feature vector which is robust to changes inthe illumination (which is little affected by the change in theillumination), and is, for example, a HOG (Histogram of Gradient)feature vector formed of a histogram of the luminance gradients in thefeature extraction region, or a feature vector formed of a histogram ofLBPs (Local Binary Patterns) obtained by binary-coding the luminancegradients in the feature extraction region.

In the following description, it is assumed that, for each pixel, ahistogram of CSLBP (Center Symmetric LBP) features, which aremodifications of LBP features, for pixels in a square feature extractionregion centered on the above-mentioned each pixel, is determined andused as the texture feature F. A histogram of CSLBPs is calculatedaccording to Non-patent reference 1.

For the calculation of CSLBP feature B for each pixel, luminance valuesof 3×3 pixels centered on the above-mentioned each pixel are needed.

For the calculation of the texture feature for each pixel, CSLBPfeatures B for all the pixels in the feature extraction region centeredon the above-mentioned each pixel are needed.

Accordingly, the texture feature cannot be calculated for a pixelpositioned at a peripheral edge part of the image.

In the following description, each of the vertical and horizontal sizesL of the feature extraction region is assumed to be 40 pixels. The sizeof the image is 640 pixels in the horizontal direction and 480 pixels inthe vertical direction, as mentioned above, and the texture feature F iscalculated for each of the pixels in a range of 600 pixels in thehorizontal direction and 440 pixels in the vertical direction, formed byremoving the parts up to 20 pixels from the edges in the horizontaldirection, and up to 20 pixels from the edges in the vertical direction.The number of pixels (600) in the horizontal direction of the region inwhich the texture feature F is calculated is denoted by M, and thenumber of pixels (440) in the vertical direction is denoted by N.

In the following description, all the pixels in the range of 600 pixelsin the horizontal direction and 440 pixels in the vertical direction maybe referred simply as “all the pixels in the image”.

Next, an exemplary method of calculation of a texture feature Fpertaining to each pixel will be described.

First, the calculation of the CSLBP feature B by the CSLBP featurecalculation unit 221 is explained with reference to FIG. 5.

The CSLBP feature B is calculated for all of the pixels in the image,except the pixels adjacent to the peripheral edges of the image (thefirst pixel as counted from the peripheral edge of the image).

FIG. 5 shows assignment of numbers to pixels which are used forcalculation of a CSLBP feature B pertaining to each pixel Pc(x,y) andare positioned in a region of 3×3 pixels centered on the above-mentionedeach pixel.

The CSLBP feature B(x,y) pertaining to a pixel Pc(x,y) is calculatedaccording to the equation (1) using luminance values of 3×3 pixelscentered on the above-mentioned each pixel.[Mathematical Expression 1]B(x,y)=s(n0−n4)*2⁰ +s(n1−n5)×2¹ +s(n2−n6)*2² +s(n3−n7)×2³   (1)

In the equation (1), n0 to n7 respectively represent luminance values ofthe pixels n0 to n7 in FIG. 5.

Also, s(a) is a threshold function, and

s(a)=1 when a>T1, and

s(a)=0 otherwise.

Since s(a) is either 0 or 1, the CSLBP feature B(x,y) is an integer, andthe range of the values which can be taken by the CSLBP feature B(x,y)is given by:0≤B(x,y)<16 ∀x,y

According to Non-patent reference 1, it is desirable that the thresholdvalue T1 is about 1% of the value which can be taken by the luminancevalue, and if the luminance value takes a value of 0 to 255, the settingis T1=3.

FIGS. 6(a) to 6(h) show examples of the results of the calculation ofthe CSLB features on the images captured under different illuminationconditions, and the luminance histograms of the images. FIGS. 6(a) and6(b) show an example of image Ga captured under illumination conditionA, and a luminance histogram of the image, FIGS. 6(c) and 6(d) showCSLBP features calculated from the image of FIG. 6(a), and a histogramof the CSLBP features, FIGS. 6(e) and 6(f) show an example of image Gbcaptured under illumination condition B, and a luminance histogram ofthe image, and FIGS. 6(g) and 6(h) show CSLBP features calculated fromthe image of FIG. 6(e), and a histogram of the CSLBP features. As shownin FIGS. 6(c), 6(d), 6(g) and 6(h), the change in the CSLBP feature issmall even when the illumination condition is changed, and the CSLBPfeature is robust to changes in the illumination condition. Accordingly,if the CSLBP features are calculated for two images, the CSLBP featureshave very close values as long as the background and the subject areunchanged, irrespective of the illumination conditions determined by theimaging/irradiating control unit 11.

When the background has changed or the subject has moved between the twoimages, the values of the CSLBP features for the pixels:Pc(xc,yc),xc∈(0,1, . . . ,W−1),yc∈(0,1, . . . ,H−1),which are included in the region where the background has changed, orthe region where the subject has moved are changed.

The texture feature calculated using the CSLBP feature in a mannerdescribed later has similar characteristics.

By calculating the texture feature F for each pixel in the image, it ispossible to grasp the feature of the texture in a local region in theimage. By comparing the features of the texture for each local region,it is possible to identify the region where the feature of the texturehas changed.

Accordingly, while it is necessary to use two images of differentillumination conditions in order for the luminance variation calculationunit 21 to obtain the luminance variation due to the change in theillumination condition, it is not necessary to use two images ofdifferent illumination conditions in order for the texture variationcalculation unit 22 to obtain the texture variation dF, but two imagesat different image capturing time points are sufficient. The two imageswith different image capturing time points may be images of the sameillumination condition.

However, in the following description, it is assumed that the texturevariation calculation unit 22 calculates the texture variation dF usingthe same images as the images used by the luminance variationcalculation unit 21. That is, it is assumed that the texture variationdF is calculated using the two images Ga, Gb of the illuminationcondition A and the illumination condition B.

The texture feature calculation unit 223 generates a plurality ofhistograms of the CSLBP features pertaining to respective pixelsincluded in the feature extraction region centered on each pixel, andgenerates, as a feature vector, a sequence of numbers obtained bysynthesizing the plurality of histograms.

For example, the texture feature calculation unit 223 includes a regiondividing unit 2211, 1st to 16th CSLBP feature reading units 2212-1 to2212-16, 1st to 16th histogram generating units 2213-1 to 2213-16, aconcatenating unit 2214, a normalizing unit 2215, and a clipping unit2216, as shown in FIG. 7.

The region dividing unit 2211 divides the feature extraction region AFcentered on each pixel into four in the vertical and horizontaldirections, as shown in FIG. 8, to generate sixteen (16) cells CA. Inthe example under consideration, the feature extraction region AFincludes 40×40 pixels, as described above, so that the size of each cellis 10×10 pixels.

The 16 cells GA are respectively allocated to; the 1st to 16th CSLBPfeature reading units 2212-1 to 2212-16.

Each CSLBP feature reading unit 2212-i (i=1 to 16) reads, from the CSLBPfeature memory 222, the CSLBP feature pertaining to each of the 10×10pixels positioned in the cell which is allocated to the CSLBP featurereading unit 2212-i.

The CSLBP feature pertaining to each of the 10×10 pixels read by eachCSLBP feature reading unit 2212-i is supplied to the correspondinghistogram generating unit 2213-i.

The histogram generating unit 2213-i generates a histogram bydetermining the occurrence frequency of each value of the CSLBP featureread by the corresponding CSLBP feature reading unit 2212-i. Since theCSLBP feature can take a value of 0 to 15, a histogram of 16 bins isgenerated.

The concatenating unit 2214 concatenates the histograms generated by the1st to 16th histogram generating units 2213-1 to 2213-16 to generate ahistogram of 16×16=256 bins, and outputs it as a feature vector of 256dimensions.

The normalizing unit 2215 normalizes the feature vector of 256dimensions output from the concatenating unit 2214, so as to make thevector length to be one.

If each of the 256 elements constituting the feature vector is denotedby v_(i)(i=0 to 255), the vector length V_(L) is given by the followingequation (2).

$\begin{matrix}\left\lbrack {{Mathematical}\mspace{14mu}{Expression}\mspace{14mu} 2} \right\rbrack & \; \\{V_{L} = \sqrt{\sum\limits_{i = 1}^{256}\; v_{i}^{2}}} & (2)\end{matrix}$

The value v_(ni) of each element after the normalization is given by thefollowing equation (3).

$\begin{matrix}\left\lbrack {{Mathematical}\mspace{14mu}{Expression}\mspace{14mu} 3} \right\rbrack & \; \\{v_{ni} = \frac{v_{i}}{V_{L}}} & (3)\end{matrix}$

The clipping unit 2216 performs a threshold processing (clipping) oneach element of the normalized vector, so that any element having avalue larger than T2 is made to have a value T2. According to Non-patentreference 1, T2 is given for example by:T2=0.2.

In the example described above, 16 CSLBP reading units are provided. Butless than 16 CSLBP reading units may be provided, and each CSLBP readingunit may be made to successively read the CSLBP features pertaining toall the pixels in two or more cells. For instance, a single CSLBPreading unit may be made to successively read the CSLBP featurespertaining to all the pixels in the 16 cells. Similarly, instead ofproviding 16 histogram generating units, less than 16 histogramgenerating units may be provided and each histogram generating unit maybe made to generate histograms for two or more cells. For instance, asingle histogram generating unit may be made to successively generatehistograms for 16 cells.

The texture feature calculation unit 223 outputs the feature vectorafter the above-mentioned clipping, as the texture feature F pertainingto the pixel positioned at the center of the feature extraction region.

The texture feature calculation unit 223 performs the above-describedprocesses on each pixel in each image, and causes the calculated texturefeature F to be stored in the texture feature memory 224.

The feature variation processing unit 225 reads, from the texturefeature memory 224, the texture features F pertaining to the two images(i.e., the image Ga obtained by the image capturing at the time point t1a, and the image Gb obtained by the image capturing at the time point t1b), and calculates a variation (texture variation) dF between theimages. The calculated texture variation dF is stored in the texturevariation memory 226. The stored texture variation dF is later suppliedto the subject extraction unit 23.

As has been described, the texture feature F is calculated for each ofthe pixels in the image, except the pixels in the peripheral edge part,and a variation in the texture feature between the two images,pertaining to the pixels at the same positions, is calculated as thevariation in the texture feature.

The variation in the texture feature F can be obtained by comparing thefeature vectors of 256 dimensions for each pixel in the two images. Thevariation in the feature vector is defined, for example, by a distancebetween the two feature vectors. As the distance between the vectors,for example, a Euclid distance, or a Manhattan distance is used.

In the following description, it is assumed that a Manhattan distance isused to calculate the variation in the feature vector. The Manhattandistance dF(t1 a,t1 b) between the feature vector F(t1 a) at the timepoint t1 a and the feature vector F(t1 b) at the time point t1 b can bedetermined by calculating the sum of the absolute values of thedifferences between corresponding elements of the two feature vectors.That is, the Manhattan distance dF(t1 a,t1 b) can be given by thefollowing equation (4).

$\begin{matrix}\left\lbrack {{Mathematical}\mspace{14mu}{Expression}\mspace{14mu} 4} \right\rbrack & \; \\{{{dF}\left( {{t\; 1\; a},{t\; 1\; b}} \right)} = {\sum\limits_{j = 1}^{J}\;{{f_{aj} - f_{bj}}}}} & (4)\end{matrix}$

In the equation (4),

f_(aj)(j=1 to J) is an element of the feature vector F(t1 a), and

f_(bj)(j=1 to J) is an element of the feature vector F(t1 b).

An element f_(aj) and an element f_(bj) having the same value of jcorrespond to each other.

J is the total number of the elements f_(aj) of the feature vector F(t1a) or the elements f_(bj) of the feature vector F(t1 b). In the presentexample, J=256.

The feature variation processing unit 225 determines the distancebetween the feature vectors for each pixel, in the manner describedabove, and causes the determined distance to be stored as the texturevariation dF in the texture variation memory 226. The stored texturevariation dF is later supplied to the subject extraction unit 23.

The texture feature F is little affected by the changes in theillumination, but when the background has changed or the subject hasmoved, the texture variation dF becomes large. For this reason, thetexture variation dF calculated between the image of the illuminationcondition A and the image of the illumination condition B can berecognized as a variation in the texture feature due to a change in thebackground, or a variation in the texture feature due to movement of asubject.

The subject extraction unit 23 extracts a subject region based on theimage G read from the image memory 14, the luminance variation dIsupplied from the luminance variation calculation unit 21, and thetexture variation dF supplied from the texture variation calculationunit 22.

As shown in FIG. 9, the subject extraction unit 23 includes a luminancevariation comparison unit 231, a texture variation comparison unit 232,a region determination unit 233, and a gate unit 234.

The luminance variation comparison unit 231 determines whether theluminance variation dI pertaining to each pixel is larger than athreshold value TI.

The texture variation comparison unit 232 determines whether the texturevariation dF pertaining to each pixel is larger than a threshold valueTF.

The region determination unit 233 determines whether each pixel belongsto a subject region (is positioned in such a region), or belongs to abackground region, or belongs to a region of false signals due tomovement of a subject or a change in the background, based on the resultof the determination by the luminance variation comparison unit 231 andthe result of the determination by the texture variation comparison unit232 for the above-mentioned each pixel.

If the result of the determination by the luminance variation comparisonunit 231 indicates that the luminance variation dI is “not larger thanthe threshold value TI”, the pixel in question is determined to belongto a background region.

If the result of the determination by the luminance variation comparisonunit 231 indicates that the luminance variation dI is “larger than thethreshold value TI”, and the result of the determination by the texturevariation comparison unit 232 indicates that the texture variation dF is“not larger than the threshold value TF”, the pixel in question isdetermined to belong to a subject region.

If the result of the determination by the luminance variation comparisonunit 231 indicates that the luminance variation dI is “larger than thethreshold value TI”, and the result of the determination by the texturevariation comparison unit 232 indicates the texture variation dF is“larger than the threshold value TF”, the pixel in question isdetermined to belong to a region of the false signals.

The gate unit 234 outputs the luminance values of the pixels which areamong the pixels constituting the image G, and which have been found bythe region determination unit 233 to belong to a subject region, as theluminance values of the pixels in the subject region.

The set of the pixels output by the gate unit 234, that is, the pixelshaving been found to belong to the subject region, constitute a subjectregion.

By such a process, the gate unit 234 outputs part of the image G readfrom the image memory 14, which coincides with the subject region, asthe result of extraction of the subject region H. The result ofextraction of the subject region H represents an image of the subjectregion H.

The reason why it is appropriate to extract a subject region by theabove-described processing is explained below.

First, the effects of a change in the illumination condition, a changein the background, and movement of the subject, on the luminancevariation and the texture variation are explained. As, shown in FIG. 10,when the illumination condition is changed, a luminance variation occursin the subject region.

When a change in the background or movement of the subject occurs, aluminance variation occurs in the region where the change or themovement occurs (for example, the part where the position of the edge ischanged due to the movement). For this reason, if a part where theluminance variation has occurred were found to be a subject regionwithout regard to any other condition, a region where the background haschanged or the subject has moved would be erroneously detected as asubject region. Therefore, according to the present invention, adetermination based on the texture variation is also made to distinguishbetween the subject region and the region where the luminance haschanged due to the change in the background or the movement of thesubject.

That is, as was explained with reference to FIGS. 6(c), 6(d), 6(g) and6(h), a change in the illumination condition does not cause any texturevariation in a subject region (FIG. 10). On the other hand, when thebackground changes or the subject moves, a texture variation occurs atthe part where the change or the movement occurs (FIG. 10). Based onthis difference, it is possible to make a distinction between thesubject region and the region where the luminance has changed due to thechange in the background or the movement of the subject.

Accordingly, it is possible to determine that the region which is withina region where the luminance variation has occurred, and in which notexture variation has occurred is a region where the luminance variationhas occurred due to a change in the illumination, and is a partconstituting the subject. On the other hand, the region where thetexture variation as well as the luminance variation has occurred can bedetermined to be a region where the luminance variation has occurred dueto the change in the background or the movement of the subject, i.e.,the region of false signals.

By using the above-described determination method, the subjectextraction unit 23 extracts the subject region based on the luminancevariation dI and the texture variation dF.

For instance, a threshold value TI is set for the magnitude of theluminance variation dI, and the luminance variation comparison unit 231determines that a pixel for which the luminance variation dI is found tobe larger than the threshold value TI is a “pixel for which a luminancevariation has occurred”. Similarly, a threshold value TF is set for themagnitude of the texture variation dF, and the texture variationcomparison unit 232 determines that a pixel for which the texturevariation dF is found to be larger than the threshold value TF is a“pixel for which a texture variation has occurred”.

The region determination unit 233 determines that a pixel for which aluminance variation is found to have occurred by the luminance variationcomparison unit 231, and no texture variation is found to have occurredby the texture variation comparison unit 232 is a pixel which belongs toa subject region.

In this case, the determination by the region determination unit 233 ismade according to the below-noted criteria.

-   (a) A pixel for which a luminance variation has occurred and a    texture variation has occurred is a pixel which belongs to a region    of false signals.-   (b) A pixel for which a luminance variation has occurred and no    texture variation has occurred is a pixel which belongs to a subject    region.-   (c) A pixel other than (a) and (b), that is, a pixel for which no    luminance variation has occurred is a pixel which belongs to a    background region.

However, since the purpose of the process is to extract a subjectregion, the determination based on (b) only may be performed, and thedeterminations based on (a) and (c) may be omitted.

The gate unit 234 reads, from the image memory 14, one of the imagesused for the luminance calculation, and outputs the luminance values ofthe pixels which are in the image having been read, and which have beendetermined to belong to a subject region by the region determinationunit 233, as the extraction result of the subject region H.

In the above-described example, the result of the determination by theluminance variation comparison unit 231 and the result of thedetermination by the texture variation comparison unit 232 for eachpixel are combined by the region determination unit 233 to make adetermination as to whether the pixel in question belongs to a subjectregion. That is, the determinations by the luminance variationcomparison unit 231, the texture variation comparison unit 232 and theregion determination unit 233 are made pixel by pixel.

Alternatively, the determinations by the luminance variation comparisonunit 231, the texture variation comparison unit 232 and the regiondetermination unit 233 may be made image by image (frame by frame). Thatis, it may be so arranged that the luminance variation comparison unit231 extracts a set of the pixels which are within the entire image andfor which a luminance variation has occurred, as a luminance variationregion, the texture variation comparison unit 232 extracts a set of thepixels which are within the entire image and for which a texturevariation has occurred, as a texture variation region, and the regiondetermination unit 233 determines the region consisting of pixels whichbelong to the luminance variation region, and which do not belong to thetexture variation region, as a subject region H.

In this case, the determination by the region determination unit 233 ismade according to the below-noted criteria.

-   (a) A region where a luminance variation has occurred and a texture    variation has occurred is a region of false signals.-   (b) A region where a luminance variation has occurred, and no    texture variation has occurred is a subject region.-   (c) A region other than (a) and (b), that is a region where no    luminance variation has occurred is a background region.

However, since the purpose of the process is to extract a subjectregion, the determination based on (b) only may be made, and thedeterminations based on (a) and (c) may be omitted.

The gate unit 234 reads, from the image memory 14, one of the imagesused for the luminance calculation, and outputs the luminance values ofthe pixels which are in the image having been read, and which belong toa region which has been found to be a subject region H by the regiondetermination unit 233, as the luminance values of the image of thesubject region H.

The threshold value TI used in the luminance variation comparison unit231 and the threshold value TF used in the texture variation comparisonunit 232 may not be fixed values and may be adaptively modified by aknown technique. For example, the threshold values may be set byapplying a percentile method, a mode method, a discriminant analysismethod or the like to a set of the luminance variations dI or a set ofthe texture variations dF.

As was described before, the images output from the imaging unit 13contain a substantial amount of noise due to dark current or chargereading noise in the imaging element, so that the luminance variationsupplied from the luminance variation calculation unit 21 and thetexture variation supplied from the texture variation calculation unit22 may contain noise.

For this reason, the subject extraction unit 23 may perform smoothingfiltering on the luminance variation dI, for example, by means of asmoothing filter, not shown, before performing the threshold processingat the luminance variation comparison unit 231. Also, when the luminancevariation comparison unit 231 performs processes on an image-by-imagebasis, smoothing filtering may be performed after the thresholdprocessing at the luminance variation comparison unit 231.

Similarly, smoothing filtering may be performed on the texture variationdF using a smoothing filter, not shown, before the threshold processingat the texture variation comparison unit 232 is performed. Also, whenthe texture variation comparison unit 232 performs processes on animage-by-image basis, smoothing filtering may be performed after thethreshold processing at this texture variation comparison unit 232.

In addition, if the region determination unit 233 performs processes onan image-by-image basis, smoothing filtering may be performed on theresults of the determination by the region determination unit 233.

As the smoothing filter, a Gaussian filter, or a median filter, forexample, may be used.

Next, examples of the operation of extracting a subject region by thesubject extraction unit 23 are explained with reference to FIGS. 11(a)to 11(e) and FIGS. 12(a) to 12(e). Here, the subject is assumed to be ahuman hand.

First, the operation of extracting a subject region in a case in whichthe subject has not moved, but the background has changed due todisappearance of a background element is explained with reference toFIGS. 11(a) to 11(e). FIGS. 11(a) to 11(e) show the captured images, theluminance variation quantity, the texture variation quantity, and theextracted subject region in a case in which, between the time point t1 aand the time point t1 b, the background changed because of adisapperance of an element B1 constituting part of the background.

FIGS. 11(a) and 11(b) show the captured images at the time points t1 aand t1 b, respectively, and also show the luminance at parts along theline BL, in the form of a graph (with the horizontal axis extending inthe x axis direction of the image coordinate system, and the verticalaxis representing the luminance). FIG. 11(c) indicates, by white, aregion where the luminance changed between the time point t1 a and thetime point t1 b, and also show the luminance variation quantity at partsalong the line BL, in the form of a graph (with the horizontal axisextending in the x-axis direction of the image coordinate system, andthe vertical axis representing the luminance variation quantity). FIG.11(d) indicates, by white, a region where the texture changed betweenthe time point t1 a and the time point t1 b, and also show the texturevariation quantity at parts along the line BL, in the form of a graph(with the horizontal axis extending in the x-axis direction of the imagecoordinate system, and the vertical axis representing the texturevariation quantity). FIG. 11(e) indicates, by white, an extractedsubject region.

The image of the time point t1 a shown in FIG. 11(a) is an image Gacaptured under the illumination condition A, and the image of the timepoint t1 b shown in FIG. 11(b) is an image Gb captured under theillumination condition B.

The subject H did not move between the time point t1 a and the timepoint t1 b. A background element B1 in the background was present at thetime point t1 a, but was not present at the time point t1 b. Forinstance, the background element B1 is the sun, which was seen at thetime point t1 a, but was hidden by a building at the time point t1 b.

The background element B1 is very bright, so that it is not affected bythe light emitted by the irradiating unit 12. The part (background part)B2 of the background other than the background element B1 issufficiently far from the irradiating unit 12 and the imaging unit 13,so that it is not affected by the light emitted by the irradiating unit12.

Referring to the captured images in FIGS. 11(a) and 11(b), the luminancevariation in FIG. 11(c), and the texture variation in FIG. 11(d), theluminance variation and the texture variation of the subject H, thebackground element B1, and the part B2 other than the background elementB1 are described.

The intensity of the light emitted by the irradiating unit 12 isstronger in the illumination condition A than in the illuminationcondition B, so that the luminance of the subject H at the time point t1a is higher than the luminance of the subject H at the time point t1 b.The subject H is therefore extracted as a region where the luminancevariation has occurred, as shown in FIG. 11(c). On the other hand, thechange in the illumination condition does not cause any veriation in thetexture feature on the subject H. Accordingly, the subject is notextracted as a region where the texture variation has occurred (FIG.11(d)).

The background element B1 differs in the brightness and the texturefeature from the background part B2, the region where the backgroundelement B1 disappeared between the time point t1 a and the time point t1b is extracted as a region where the luminance variation has occurred(FIG. 11(c)), and also as a region where the texture variation hasoccurred (FIG. 11(d)).

No change occurred in the luminance and the texture between the timepoint t1 a and the time point t1 b in the background part B2 (to beaccurate, the region of the background at the time point t1 b, which isother than the region where the background element B1 existed at thetime point t1 a), so that the background part B2 is not extracted as aluminance variation region, nor as a texture variation region (FIG.11(c) and FIG. 11(d)).

By recognizing a region where a luminance variation has occurred, and notexture variation has occurred as a subject region, and recognizing aregion where a luminance variation has occurred and, also, a texturevariation has occurred as a region of false signals, it is possible todistinguish the subject region from the region where the backgroundelement B1 has disappeared, and to extract the subject region H only, asshown in FIG. 11(e).

Next, the operation of extracting a subject region in a case in whichthe subject has moved is explained with reference to FIGS. 12(a) to12(e). FIGS. 12(a) to 12(e) show the captured images, the luminancevariation quantity, the texture variation quantity, and the extractedsubject region in a case in which, between the time point t1 a and thetime point t1 b, the subject H moved and, also, the background changedbecause of movement of an element B1 constituting part of thebackground.

FIGS. 12(a) and 12(b) show the captured images at the time points t1 aand t1 b, respectively, and also show the luminance at parts along theline BL, in the form of a graph (with the horizontal axis extending inthe x axis direction of the image coordinate system, and the verticalaxis representing the luminance). FIG. 12(c) indicates, by white, aregion where the luminance changed between the time point t1 a and thetime point t1 b, and also show the luminance variation quantity at partsalong the line BL, in the form of a graph (with the horizontal axisextending in the x-axis direction of the image coordinate system, andthe vertical axis representing the luminance variation quantity). FIG.12(d) indicates, by white, a region where the texture changed betweenthe time point t1 a and the time point t1 b, and also show the texturevariation quantity at parts along the line BL, in the form of a graph(with the horizontal axis extending in the x-axis direction of the imagecoordinate system, and the vertical axis representing the texturevariation quantity). FIG. 12(e) indicates, by white, an extractedsubject region.

The image of the time point t1 a shown in FIG. 12(a) is an image Gacaptured under the illumination condition A, and the image of the timepoint t1 b shown in FIG. 12(b) is an image Gb captured under theillumination condition B.

The subject H moved to the right between the time point t1 a and thetime point t1 b.

Like the subject H, a background element B1 in the background moved tothe right between the time point t1 a and the time point t1 b.

The background element B1 is very bright, so that it is not affected bythe light emitted by the irradiating unit 12. The part B2 of thebackground other than the background element B1 is sufficiently far fromthe irradiating unit 12 and the imaging unit 13, so that it is notaffected by the illuminating light from the irradiating unit 12.

Referring to the captured images in FIGS. 12(a) and 12(b), the luminancevariation in FIG. 12(c), and the texture variation in FIG. 12(d), theluminance variation and the texture variation of the subject H, thebackground element B1, and the part B2 other than the background elementB1 are described.

The intensity of the light emitted by the irradiating unit 12 isstronger in the illumination condition A than in the illuminationcondition B, so that the luminance of the subject H at the time point t1a is higher than the luminance of the subject H at the time point t1 b.The subject H is therefore extracted as a region where the luminancevariation has occurred in the luminance variation calculation resultshown in FIG. 12(c). Also, because the subject H moved, the boundaryparts on the left and right sides of the subject (the part which changedfrom the subject to the background, and the part which changed from thebackground to the subject) are extracted as regions where the luminancevariation has occurred (FIG. 12(c)). That is, in the boundary part onthe left side of the subject region (the part which changed from thebackground to the subject), a luminance variation is detected because ofthe difference between the brightness of the subject under theillumination condition B and the brightness of the background part B2,while in the boundary part on the right side of the subject region (thepart which changed from the subject to the background), a luminancevariation is detected because of the difference between the brightnessof the subject under the illumination condition A and the brightness ofthe background part B2.

Also, because the subject H moved, the boundary parts on the left andright sides of the subject H (the part which changed from the backgroundto the subject, and the part which changed from the subject to thebackground) are extracted as texture variation regions, because of thedifference in the texture feature between the background part B2 and thesubject H (FIG. 12(d).

The background element B1 differs in the brightness and the texturefeature from the background part B2, and since the background element B1moved between the time point t1 a and the time point t1 b, the boundaryparts on the left and right sides of the background element B1 areextracted as luminance Variation regions (FIG. 12(c)), and also astexture variation regions (FIG. 12(d)).

No change occurred in the luminance and the texture between the timepoint t1 a and the time point t1 b in the background part B2 (to beaccurate, the part which was not part of the background element B1 atthe time point t1 a, nor at the time point t1 b), so that the backgroundpart B2 is not extracted as a luminance variation region, nor as atexture variation region (FIG. 12(c), FIG. 12(d)).

By recognizing a region where a luminance variation has occurred and notexture variation has occurred as a subject region, and recognizing aregion where a luminance variation has occurred and, also, a texturevariation has occurred as a region of false signals, it is possible todistinguish the subject region from the boundary parts of the subject H(the parts where the luminance has changed), and the boundary parts ofthe background element B1 (the parts where the luminance has changed),and to extract the subject region H only, as shown in FIG. 12(e).

As has been described, according to the present invention, it ispossible to distinguish the subject region from the background regionand also from the region of false signals, based on the luminancevariation and the texture variation, thereby extracting the subjectregion with a high accuracy.

If there are differences in the patterns, the unevenness, or thereflectivity of the surface of the subject or the background element,(objects or the like constituting the background) so that the luminanceis not uniform, movement of the subject or the background element causeschanges in the texture feature not only in the vicinity of the regionboundary, but also in the interior of the subject region or thebackground region. However, according to the present embodiment, atexture feature is calculated for a feature extraction region definedfor each pixel. Instead of just each pixel and its adjacent pixels,pixels in a greater range are used for the determination, so that theeffects of the movement of the subject or the background element on thetexture feature variation in the subject region or the background regionis reduced. The variation in the texture feature at the region boundaryis relatively large compared with the variation within the region,unless the texture of the subject and the texture of the backgroundelement are similar to each other. For this reason, by performing athreshold processing on the texture variation, only the texturevariation at the region boundary can be extracted, by distinguishing itfrom the texture variation within the region.

Next, a processing procedure in the imaging apparatus of the firstembodiment will be described with reference to FIG. 13.

The process shown in FIG. 13 is performed frame by frame, i.e., once aframe, period.

First, in a step ST1, prior to the image capturing in each frame period,the imaging/irradiating control unit 11 generates the control signalsC11 a and C11 b, and outputs them to the irradiating unit 12 and theimaging unit 13.

The imaging/irradiating control unit 11 controls the irradiating unit12, causing it to perform the irradiation with the illuminationcondition A and the irradiation with the illumination condition Balternately frame by frame. For instance, the control is so made thatthe irradiation with the illumination condition A is performed for theodd-numbered frames, and the irradiation with the illumination conditionB is performed for the even-numbered frames. Such control over theillumination condition can be regarded as one taking two frame periodsas one cycle, and causing the illumination with the illuminationcondition A to be performed in the first frame in each cycle, andcausing the illumination with the illumination condition B to beperformed in the second frame.

Next, in a step ST2, the irradiating unit 12 performs irradiation basedon the control signal C11 a from the imaging/irradiating control unit11. By this irradiation, an illumination condition corresponding to thecontrol signal C11 a is generated.

Next, in a step ST3, image capturing is performed under the illuminationcondition generated by the step ST2, to obtain a captured image, and thecaptured image is stored in the image memory 14.

Next, in a step ST4, the luminance feature quantity calculation unit 211in the luminance variation calculation unit 21 reads the latest imagefrom the image memory 14, and determines the luminance feature quantityIm of the image having been read. The calculated luminance featurequantity Im is stored in the luminance feature quantity memory 212.

In a step ST5 performed in parallel with the step ST4, the CSLBP featurecalculation unit 221 in the texture variation calculation unit 22 readsthe latest image from the image memory 14, and calculates the CSLBPfeature B of each pixel in the image having been read. The calculatedCSLBP feature B is stored in the CSLBP feature memory 22.

Next, in a step ST6, the texture feature calculation unit 223 in thetexture variation calculation unit 22 reads the CSLBP featurespertaining to respective pixels in the latest image from the CSLBPfeature memory 222, and calculates the texture feature F pertaining toeach pixel in the same image based on the CSLBP features B having beenread. The calculated texture feature F is stored in the texture featurememory 224.

Next, in a step ST7, the illumination condition determination unit 213in the luminance variation calculation unit 21 reads the luminancefeature quantity Im of the latest image and the luminance featurequantity Im of the image one frame before, from the luminance featurequantity memory 212, compares them, decides which of the images has thelarger luminance feature quantity Im, and determines, based on theresult of the decision, which of the images is an image Ga of theillumination condition A, and which is an image Gb of the illuminationcondition B. The result of this determination, CNa, is supplied to thedifference calculation unit 215.

After the step ST7, the process proceeds to steps ST8 and ST9.

In the step ST8, the difference calculation unit 215 in the luminancevariation calculation unit 21 reads, from the image memory 14, thelatest image (image of the current frame), and the image one framebefore, and calculates the luminance difference of each pixel. Thecalculated luminance difference is stored as the luminance variation dIin the luminance variation memory 216, and is later supplied to thesubject extraction unit 23.

In the calculation of the difference, based on the result of theidentification in the step ST7, the luminance value of each pixel in theimage Gb of the illumination condition B (the image which was found tohave the smaller luminance feature quantity Im in the step ST7) issubtracted from the luminance value of the pixel at the same position inthe image Ga of the illumination condition A (the image which was foundto have the larger luminance feature quantity Im in the step ST7). Ifthe result of the subtraction is a negative value, the luminancedifference is treated as being zero.

The calculated luminance variation dI of each pixel is stored in theluminance variation memory 216.

In the step ST9, the feature variation processing unit 225 in thetexture variation calculation unit 22 calculates the texture variationdF of each pixel between the latest image and the image one framebefore. The calculated texture variation dF of each pixel is stored inthe texture variation memory 226.

After the steps ST8 and ST9, the process proceeds to a step ST10.

In the step ST10, the subject extraction unit 23 extracts a subjectregion based on the luminance variation dI and the texture variation dF.

In the extraction of the subject region, the determination as to whethereach pixel belongs to the subject region or not may be made pixel bypixel, or frame by frame. Below, a procedure in which the determinationis made on a pixel-by-pixel basis is first described with reference toFIG. 14, after which a procedure in which the determination is made on aframe-by-frame basis is described with reference to FIG. 15.

In the procedure shown in FIG. 14, first, in a step ST11, one of thepixels in the image to be processed is selected. Here, the pixels in theimage to be processed means pixels in the part other than the peripheraledge part of the image. This also applies to the following description.Moreover, all the pixels positioned in the part other than theperipheral edge part may be referred to simply as “all the pixels in theimage”.

The selection may be in the order from the upper left to the lower rightof the image.

Next, in a step ST12, the luminance variation dI pertaining to theselected pixel (pixel of interest) is read from the luminance variationmemory 216. Next, in a step ST13, whether the read luminance variationdI is larger than the threshold value TI (that is, whether the luminancevariation is “present” or “absent” is determined.

If the luminance variation dI is not larger than the threshold value TI(if the luminance variation is “absent”), the pixel of interest isdetermined to belong the background region (ST14).

If the luminance variation dI is larger than the threshold value TI, theprocess proceeds to a step ST15. In the step ST15, the texture variationdF pertaining to the pixel of interest is read from the texturevariation memory 226.

Next, in a step ST16, whether the read texture variation dF is largerthan the threshold value TF (whether the texture variation is “present”or “absent”) is determined.

If it is found that the texture variation dF is not larger than thethreshold value TF (it is found that the texture variation is “absent”),the pixel of interest is determined to belong to the subject region(ST17).

If it is found that the threshold value TF is exceeded (it is found thatthe “texture variation” is present), the pixel of interest is determinedto be in a part where the subject has moved, or the background haschanged, that is a region of false signals (ST18). Here, the part wherethe subject has moved means a part which was a part of the subject oneframe before, and was a part of the background in the latest frame, or apart which was a part of the background one frame before, and was a partof the subject in the latest frame. According to the present invention,these parts are not recognized as parts of the subject.

Next, in a step ST19, whether all the pixels in the image to beprocessed have been selected, that is whether the processes of the stepsST12 to ST18 have been performed for all the pixels in the image to beprocessed is determined. If there is any pixel yet to be selected, theprocess returns to the step ST11.

If all the pixels have been selected, the process proceeds to a stepST20.

In the step ST20, a set of the pixels which were determined to belongthe subject region in the step ST17 is output as the extraction resultof the subject region H.

Since the purpose of the process is to extract the subject region, theprocesses of the steps ST14 and ST18 may be omitted. In this case, ifthe result of the determination in the step ST13 is NO, or if the resultof the determination in the step ST16 is YES, the process proceedsdirectly to the step ST19.

In the procedure shown in FIG. 15, first, in a step ST21,

the pixels in the image are successively selected, the luminancevariation dI of the selected pixel is read from the luminance variationmemory 216, and whether the read luminance variation dI is larger thanthe threshold value TI (that is, whether the luminance variation is“present” or “absent”) is determined. A set of the pixels for which theluminance variation is found to be “present” is extracted as a luminancevariation region.

In a next step ST22, the pixels in the image are successively selected,the texture variation dF of the selected pixel is read from the texturevariation memory 226, and whether the read texture variation dF islarger than the threshold value TF (that is whether the texturevariation is “present” or “absent”) is determined. A set of the pixelsfor which the texture variation is found to be “present” is extracted asthe texture variation region.

In a next step ST23, a set of the pixels which belong to the “luminancevariation region” and which do not belong to the “texture variationregion” is extracted as the “subject region”.

In the next step ST20, a set of the pixels which were determined tobelong to the subject region extracted in the step ST23 is output as theextraction result of the subject region H.

In the above example, the texture feature F is calculated for all thepixels in the image, and is stored in the texture feature memory 224,and, thereafter, the texture features F for the same pixel in the twoimages are read and the texture variation dF is calculated.Alternatively, it may be so arranged that when the texture feature F iscalculated for each pixel in each image, the texture variation dF iscalculated from the texture feature F that has just been calculated, andthe texture feature F (stored for instance in the texture feature memory224) calculated previously for the pixel at the same position in theimage one frame before. With such an arrangement, the time from theimage capturing to the calculation of the texture variation dF can beshortened.

When the process is performed in such a procedure, a texture variationcalculation unit 22 a shown in FIG. 16 is used in place of the texturevariation calculation unit 22 shown in FIG. 4. The texture variationcalculation unit 22 a shown in FIG. 16 includes a CSLBP featurecalculation unit 221, a CSLBP feature memory 222, a texture featurecalculation unit 223 a, a texture feature memory 224, a featurevariation processing unit 225 a, and a texture variation memory 226.

Among the components shown in FIG. 16, the CSLBP feature calculationunit 221, the CSLBP feature memory 222, the texture feature memory 224,and the texture variation memory 226 are similar to those shown in FIG.4.

The texture feature calculation unit 223 a successively calculates thetexture feature F pertaining to each pixel in each image read from theimage memory 14, and causes it to be stored in the texture featurememory 224, and supplies it to the feature variation processing unit 225a.

The feature variation processing unit 225 a receives the texture featureF calculated by the texture feature calculation unit 223 a for the pixelbeing selected (pixel of interest) in the latest image, receives thetexture feature F pertaining to the pixel at the same position in theimage one frame before, stored in the texture feature memory 224, andcalculates a variation between the two texture features, and causes theresult of the calculation as the texture variation dF in the texturevariation memory 226. The stored texture variation dF is later suppliedto the subject extraction unit 23.

A processing procedure to be followed in this case is shown in FIG. 17and FIG. 18.

In FIG. 17 and FIG. 18, the same reference characters as in FIG. 13denote identical or similar processes.

That is, the processes in the steps ST1 to ST4, ST5, and ST7 in FIG. 17are identical to those in FIG. 13.

After the step ST7, the process proceeds to a step ST11 a.

In the step ST11 a, one of the pixels in the image to be processed isselected.

After the step ST11 a, the process proceeds to a step ST8 a and a stepST6 a.

In the step ST8 a, the difference calculation unit 215 in the luminancevariation calculation unit 21 reads luminance values of the selectedpixel (pixel of interest) in the latest image and the pixel at the sameposition as the pixel of interest in the image one frame before, storedin the image memory 14, calculates the luminance difference, and causesthe calculated difference to be stored as a luminance variation dI inthe luminance variation memory 216. The stored luminance variation dI islater supplied to the subject extraction unit 23.

In the step ST6 a, the texture feature calculation unit 223 a calculatesthe texture feature F for the selected pixel (pixel of interest), causesthe calculated texture feature F to be stored in the texture featurememory 224, and supplies it to the feature variation processing unit 225a.

In a step ST9 a, the feature variation processing unit 225 a calculatesthe texture variation dF pertaining to the pixel of interest, from thetexture feature pertaining F to the pixel of interest, supplied from thetexture feature calculation unit 223 a (calculated in the step ST6 a),and the texture feature F pertaining to the pixel at the same positionin the image one frame before, stored in the texture feature memory 224,and causes the calculated texture variation dF to be stored in thetexture variation memory 226. The stored texture variation is latersupplied to the subject extraction unit 23.

The steps ST6 a, ST8 a, and ST9 a in FIG. 17 differ from the steps ST6,ST8, and ST9 in FIG. 13 in that the processes in the steps ST6 a, ST8 a,and ST9 a are for the selected pixels whereas the processes in the stepST6, ST8, and ST9 are for all the pixels constituting the image.

After the steps ST8 a and ST9 a, the process proceeds to the step ST13.

In the step ST13, whether the luminance variation dI calculated in thestep ST8 a is larger than the threshold value TI (that is whether theluminance variation is “present” or “absent”) is determined.

If the threshold value TI is not exceeded (if the luminance variation is“absent”), the pixel of interest is determined to belong to thebackground region (ST14).

If the threshold value TI is exceeded, the process proceeds to the stepST16. In the step ST16, whether the texture variation dF calculated inthe step ST9 a is larger than the threshold value TF (whether thetexture variation is “present” or “absent”) is determined.

If it is found that the threshold value TF is not exceeded (if thetexture variation is found to be “absent”), the pixel of interest isdetermined to belong to a subject region (ST17).

If it is found that the threshold value TF is exceeded (if the texturevariation is found to be “present”), the pixel of interest is determinedto belong to a region of false signals (ST18).

Next, in the step ST19, whether all the pixels in the image to beprocessed have been selected, that is whether the processes of the stepsST6 a, ST8 a, ST9 a, ST13, ST14, and ST16 to ST18 have been performedfor all the pixels is determined. If there is any pixel yet to beselected, the process returns to the step ST11 a.

If all the pixels have been selected, the process proceeds to the stepST20.

In the step ST20, a set of the pixels which were determined to belong tothe subject region in the step ST17 is output as the extraction resultof the subject region.

As was also described with reference to FIG. 13, the processes of thesteps ST14 and ST18 may be omitted. In this case, in the case of “NO” inthe step ST13, or in the case of “YES” in the step ST16, the processproceeds directly to the step ST19.

In the procedure shown in FIG. 17 and FIG. 18, the processes of the stepST6 a and ST9 a are performed before the step ST13. Alternatively, itmay be so arranged that if the result of the determination in the stepST13 is “YES”, then the processes of the step ST6 a and the step ST9 aare performed, and thereafter the process proceeds to the step ST16.

As has been described, according to the imaging apparatus of the presentembodiment, the imaging/irradiating control unit 11 outputs the controlsignal C11 a for controlling the illumination condition and the controlsignal C11 b for controlling the imaging condition, the irradiating unit12 generates two different illumination conditions based on the controlsignal C11 a, the imaging unit 13 captures the image of the subjectunder the two different illumination conditions, the luminance variationcalculation unit 21 calculates the luminance variation dI between thetwo images captured under the different illumination conditions, thetexture variation calculation unit 22 calculates the texture variationdF between the two images captured at the different time points, and thesubject extraction unit 23 distinguishes the luminance variation whichhas occurred in the subject due to the illumination change, from theluminance variation due to the background change or the subjectmovement, based on the luminance variation dI and the texture variationdF.

Accordingly, the extraction of the subject region can be achieved with ahigh accuracy.

According to the imaging apparatus of the present embodiment, payingattention to a characteristic in which a luminance variation and atexture variation occur in a region where the background change or thesubject movement occurs, whereas only a luminance variation due to theillumination change occurs in a situation in which no background changenor subject movement occurs, the region where the luminance variation islarge and the texture variation is small is recognized as a subjectregion, making it possible to remove the effects of the backgroundchange and the subject movement, and to extract the subject region witha high accuracy.

Also, two illumination conditions, namely, a first illuminationcondition in which the light is emitted with a first light emittingintensity, and a second illumination condition in which the light isemitted with a second light emitting intensity which is smaller than thefirst light emitting intensity are used. Accordingly, the subject regioncan be extracted using the luminance variation and the texturevariation, by a simple configuration and processes.

Furthermore, by making the illumination conditions for capturing theimages used for the calculation of the luminance variation by theluminance variation calculation unit 21, and the illumination conditionsfor capturing the images used for the calculation of the texturevariation by the texture variation calculation unit 22 to be identicalwith each other, it is possible to determine whether the cause of theluminance variation is the illumination change, or either of thebackground change and the subject movement, making it possible toextract the subject region with a high accuracy.

In the imaging apparatus of the present embodiment, the irradiationdistribution with which the illuminating light is appropriatelyirradiated on the subject which is desired to be extracted can be usedas one of the illumination conditions, and by doing so, it is possibleto obtain the images which are suitable for the calculation of theluminance variation and the texture variation regardless of the positionand the size of the subject.

Moreover, the intensity with which the illuminating light of such alight quantity suitable for the reflectivity of the subject which isdesired to be extracted is irradiated can be used as one of theillumination conditions, and by doing so, it is possible to obtainimages which are suitable for the calculation of the luminance variationand the texture variation.

Furthermore, both of the above-mentioned two conditions mentioned abovecan be used as the illumination conditions, and by doing so, it ispossible to obtain images suitable for the calculation of the luminancevariation and the texture variation, depending on the position, the sizeand the reflectivity of the subject.

In the present embodiment, two illumination conditions (illuminationcondition A and illumination condition B) which are related to theintensity of the emitted light are used to extract the subject region.However, two illumination conditions related to the irradiationdistribution of the illuminating light on the subject may be used. Forinstance, if the position of the subject in the captured image is fixedto a certain degree, the illumination condition may be so set that theilluminating light is properly distributed on to the position of thesubject. Also, the irradiation distribution may be set depending on thechange in the local ambient light. The irradiation distribution may beadjusted by changing the irradiation angles of the LEDs, or by selectingthe LEDs to be turned on to emit light, among the plurality of LEDs.

In the above-described embodiment, the calculations of the luminancevariation and the texture variation are performed between two images.However, the present invention is not limited to this scheme, and thecalculations of the luminance variation and the texture variation may beperformed between three or more images. In summary, it is satisfactoryif the calculations of the luminance variation and the texture variationare performed between a plurality of images. When the calculation of theluminance variation is performed between three or more images, imagescaptured under mutually different illumination conditions are used. Whenthe calculation of the texture variation is performed between three ormore images, images captured at mutually different time points are used.The images captured at mutually different time points may be imagescaptured under mutually different illumination conditions. Accordingly,they may be the same as the images which are used for the calculation ofthe luminance variation.

Increase of the number of images used for the calculation of theluminance variation and the texture variation leads to increase in thememory amount or the processing load, i.e., increase in the requiredhardware resources and the processing time. These points must be takeninto account in determining the number of the images used for thecalculation of the luminance variation and the texture variation.

In the present embodiment, the luminance value of the image Gb of theillumination condition B is subtracted from the luminance value of theimage Ga of the illumination condition A for generating the differenceimage. The present invention is not limited to this. For instance anabsolute value of the luminance difference may be used.

In the present embodiment, the texture variation is calculated using theimages which are also used for the calculation of the luminancevariation. The present invention is not limited to this. If two imagescaptured at different time points are used, it is possible to detect thebackground change or the subject movement by calculating the texturevariation, without regard to the illumination condition.

However, it is necessary to use images captured at timings, close enoughto the images used for the calculation of the luminance variation.

In the present embodiment, the texture variation dF is calculated forall the pixels in the image. However, based on the calculated luminancevariation dI, the texture variation dF may be calculated only for thepixels, which are in a region where the luminance variation has occurred(the luminance variation dI is larger than the threshold value TI). Inthis case, the amount of processing of the texture variation dF isreduced.

In the above-described embodiment, a square feature extraction regioncentered on the pixel of interest is used for the calculation of thetexture feature. The shape of the feature extraction region is notlimited to square. That is, the shape of the feature extraction regionmay be of a circle, or any polygon, not being limited to a rectangle.However, in any case, it is desirable that the feature extraction regionis centered on the pixel of interest. Also, the size of the featureextraction region is not limited to the above example.

In the above-described embodiment, the texture feature and the texturevariation are calculated for each of all the pixels in the image, otherthan the peripheral edge part.

However, the present invention is not limited to this scheme. Forinstance, the texture feature and the texture variation may becalculated only for the pixels which are at positions at a predeterminedinterval in the image, other than the peripheral edge part. Forinstance, a lattice pattern may be drawn on the image as shown in FIG.19, and the texture feature and the texture variation may be calculatedonly for such pixels which are positioned at the lattice points.

In the example shown in FIG. 19, the lattice is a square lattice, andthe intervals in the vertical direction and in the horizontal directionbetween the lattice points are denoted by a reference character D.

When the texture feature and the texture variation are calculated onlyfor the pixels which are positioned at the lattice points as shown inFIG. 19, with regard to the pixels at positions other than the latticepoints, the texture feature and the texture variation calculated for thepixel at the closest lattice point may be used as the texture featureand the texture variation of the pixels at the positions other than thelattice points.

The example in which the texture feature and the texture variation arecalculated for all the pixels, described in the above embodiment,corresponds to a case in which D is one pixel in FIG. 19.

The pixels for which the texture feature and the texture variation arecalculated are not limited to those positioned at the lattice points inthe image.

The intervals between the pixels, and hence the density of the pixelsfor which the texture feature and the texture variation are calculatedneed not be uniform throughout the image. For instance, when theposition at which the subject is present in the image is known inadvance, the density of the pixels for which the texture feature and thetexture variation are calculated may be made high in the region nearsuch a position, and may be made low in other regions.

In the above-described embodiment, the feature extraction region isdivided into 4×4 cells for the calculation of the texture feature.However, the number of cells is not limited to this. For instance, byreducing the size of each cell, and increasing the number of cells inthe feature extraction region, more detailed texture feature can beobtained. But the processing load is increased because of the increasein the number of dimensions of the feature vector. The number of cellsand the size of each cell should be determined taking account of thesepoints and the characteristics of the apparatus to which the presentinvention is applied.

In the above-described embodiment, the illumination conditions of theimages read from the image memory 14 are identified from the luminancefeature quantities (e.g., the luminance mean values) of the images. Thepresent invention is not limited to this scheme.

For instance, it may be so arranged that, like the control signal C11 asupplied to the irradiating unit 12, a signal indicating theillumination condition (e.g., the light emitting intensity) is suppliedto the imaging unit 13, the imaging unit 13 identifies the illuminationcondition based on the above-mentioned signal, adds information(appendix information) indicating the illumination condition whencausing the captured image to be stored in the image memory 14, and theluminance variation calculation unit 21 identifies the illuminationconditions of the images based on the appendix information, when readingthe images from the image memory 14.

In the above-described embodiment, one of the images used for theluminance calculation is read from the image memory 14, and the part ofthe read image which coincides with the subject region extracted by thesubject extraction unit 23 is taken as the extraction result (subjectimage). However, the present invention is not limited to this.

For instance, such part of the image consisting of the luminancevariations calculated by the luminance variation calculation unit 21(the image in which the luminance variations for all the pixels arearranged in the same manner as the corresponding pixels), i.e., theluminance difference image, which coincides with the subject regionextracted by the subject extraction unit 23 may be taken as theextraction result (subject image). The use of part of the luminancedifference image as the subject image is advantageous because theluminance difference image is free from the effects of the ambientlight.

Furthermore, in the above-described embodiment, the luminance variationcalculation unit 21 reads, from the image memory 14, the latest imageand the image one frame before, and calculates the luminance variationof each pixel between the two images. As a result, the luminancevariation is calculated between the image Ga of the illuminationcondition A and the subsequent image Gb of the illumination condition B,and the luminance variation is also calculated between the image Gb ofthe illumination condition B and the subsequent image Ga of theillumination condition A. This is also true for the texture variation.

Alternatively, the luminance variation calculation unit 21 may be soconfigured that the luminance variation is calculated between the imageGa of the illumination condition A, and the image Gb of the illuminationcondition B, of the immediately succeeding frame, and the luminancevariation is not calculated between the image Ga of the illuminationcondition A, and the image Gb of the illumination condition B, of theimmediately preceding frame. In other words, the arrangement may be suchthat the luminance variation is calculated only when the latest image isan image Gb of the illumination condition B, and the immediatelypreceding image is an image Ga of the illumination condition A.Similarly, the texture variation calculation unit 22 may be soconfigured to calculate the texture variation only when the latest imageis an image Gb of the illumination condition B, and the immediatelypreceding image is an image Ga of the illumination condition A.

In such a case, in place of the procedure shown in FIG. 13, a procedureshown in FIG. 20 is used. In FIG. 20, a step ST7 a is performed in placeof the step ST7 in FIG. 13. In the step ST7 a, whether the latest imageis an image Gb of the illumination condition B is determined, and if theresult of the determination is “NO”, the process in the particular frameends. If the result of the determination is “YES”, the processes of thesteps ST8 and ST9 and the subsequent steps are performed.

Alternatively, the luminance variation and the texture variation may becalculated only when the latest image is an image Ga of the illuminationcondition A and the immediately preceding image is an image Gb of theillumination condition B.

Second Embodiment

FIG. 21 is a block diagram showing the configuration of an imagingapparatus according to a second embodiment of the present invention. Theimaging apparatus shown in FIG. 21 is generally identical to the imagingapparatus shown in FIG. 1, but is different in that a target valuecalculation unit 24 is added, and an imaging/irradiating control unit 11b is provided in place of the imaging/irradiating control unit 11.Reference characters identical to those in FIG. 1 denote identical orcorresponding components, so that their description is omitted.

In the imaging apparatus shown in FIG. 1, the illumination condition andthe imaging conditions are fixed. In the imaging apparatus according tothe present embodiment, the target value calculation unit 24 adjusts atleast one of the illumination condition and the imaging condition basedon the texture variation, the extraction result of the subject region,and the luminance feature quantities of the captured images, to improvethe accuracy of the extraction of the subject region despite changes inthe ambient light and the background.

The adjustment in the illumination condition or the imaging condition isperformed by adjusting a target value related to the illuminationcondition or the imaging condition, and performing control such that theactual value related to the illumination condition or the imagingcondition are made to be equal to the target value.

The imaging/irradiating control unit 11 b receives a target value signalQ related to the illumination condition and the imaging condition fromthe target value calculation unit 24. Based on the target value signalQ, the imaging/irradiating control unit 11 b generates the controlsignal C11 a, and outputs the control signal C11 a to the irradiatingunit 12, and generates the control signal C11 b, and outputs the controlsignal C11 b to the imaging unit 13.

The target value calculation unit 24 reads the images captured under thetwo illumination conditions, receives the texture variation dF from thetexture variation calculation unit 22, also receives the extractionresult of the subject region from the subject extraction unit 23,calculates, based on these inputs, a target value of at least one of theillumination condition and the imaging condition, and outputs the targetvalue signal Q representing the calculated target value to theimaging/irradiating control unit 11 b.

The target value signal Q output from the target value calculation unit24 to the imaging/irradiating control unit 11 b is used for thegeneration of the control signals C11 a and C11 b in theimaging/irradiating control unit 11 b. The target value signal Qrepresents the target value of, for example, at least one of the shapeof the irradiation distribution of the illuminating light, the lightemitting intensity of the illuminating light, the light emitting time ofthe illuminating light, the exposure time of the imaging unit, theaperture of the imaging unit and the gain of the imaging unit. However,the target value signal Q may not necessarily represent a target valueof a numerical value (an absolute value) which directly represents theshape of the irradiation distribution of the illuminating light, thelight intensity of the illuminating light, the light, emitting time ofthe illuminating light, the exposure time: of the imaging unit, theaperture of the imaging unit or the gain of the imaging unit, but mayrepresent a target value of a relative value, or may be a code of theabsolute value or the relative value. In the following description, thetarget value signal is assumed to represent a target value of the lightemitting intensity of the illuminating light.

The target value calculation unit 24 calculates the above-mentionedtarget value such that images suitable for the calculation of theluminance variation dI in the luminance variation calculation unit 21and the calculation of the texture variation dF in the texture variationcalculation unit 22 are obtained by the imaging unit 13.

The target value calculated by the target value calculation unit 24 isstored in an internal target value memory 244 (to be described later).

When the imaging/irradiating control unit 11 b controls the illuminationcondition or the imaging condition, the target value calculation unit 24reads the target value stored in the target value memory 244, andsupplies the target value signal Q representing the read target value,to the imaging/irradiating control unit 11 b.

As shown in FIG. 22, the target value calculation unit 24 includes anin-subject variation region extraction unit 241, an area determinationunit 242, a target value adjusting unit 243, and the target value memory244.

The in-subject variation region extraction unit 241 extracts a regionRLH which is positioned within the subject region H output from thesubject extraction unit 23, and in which the texture variation dF isrelatively large.

Whether the texture variation is relatively large is determined based onthe texture variation dF output from the texture variation calculationunit 22.

For instance, the region in which the above-mentioned texture variationdF is larger than a threshold value TFa is determined to be a texturevariation region RLH, and the region other than the texture variationregion is determined to be a no-variation region RLL.

For the purpose of distinction from the region (texture variationregion) in which the texture variation dF is determined to be largerthan the threshold value TF by the subject extraction unit 23, theregion in which the texture variation dF is determined to be larger thanthe threshold value TFa by the in-subject variation region extractionunit 241 is called an in-subject variation region.

For example, the threshold value TFa is so set as to satisfy TFa<TF. Bysetting the threshold value TFa in this manner, it is possible toextract, from the subject region (in which the luminance variation dI isnot smaller than the threshold value TI and the texture variation dF isnot larger than the threshold value TF) extracted using the thresholdvalue TF by the subject extraction unit 23, a region (an in-subjectvariation region) RLH in which the texture variation dF is larger thananother threshold value TFa.

The area determination unit 242 determines whether the area of thein-subject variation region RLH, for example, the number of pixelsincluded in the region RLH, extracted by the in-subject variation regionextraction unit 241 is not smaller than the threshold value TAa, andoutputs the result of the determination.

The texture variation dF in the subject region represents a texturevariation between images of different illumination conditions, and thein-subject variation region RLH can be recognized a region where thetexture feature has varied due to the change in the illuminationcondition.

Generally, variations in the texture feature due to changes in theillumination conditions are small. If, despite such a nature, thetexture feature varies to a certain degree or more due to the change inthe illumination condition, that is, the area of the in-subjectvariation region RLH is not smaller than the threshold value TAa, it isdetermined that adjustment (alteration) of the illumination condition orthe imaging condition is necessary.

When the area determination unit 242 determines that the area of thein-subject variation region RLH is not smaller than the threshold valueTAa, the target value adjusting unit 243 calculates a new target valueresponsive to the result of the determination.

The new target value is so determined that the difference in the lightemitting intensity between the two illumination conditions A and B isreduced. For instance, if the texture variation dF was calculated usingthe image captured at the time point t1 a under the illuminationcondition A (light emitting intensity ϕA(t1 a)), and the image capturedat the time point t1 b under the illumination condition B (lightemitting intensity ϕB(t1 b)), then the target value ϕA(t2 a) of thelight emitting intensity at the time point t2 a for the next imagecapturing under the illumination condition A, and the target value ϕB(t2b) of the light emitting intensity at the time point t2 b for the nextimage capturing under the illumination condition B are respectivelyadjusted according to the equations (5) and (6). By adjusting the targetvalues in this way, the difference between the target values of thelight emitting intensities for the two illumination conditions A and Bis reduced.ϕA(t2a)=ϕA(t1a)−ΔΦa  (5)ϕB(t2b)=ϕB(t1b)−ΔΦb  (6)

In the equations (5) and (6), Δϕa denotes an amount of adjustment.

The target values of the light emitting intensities are thus adjustedand the adjusted target values are supplied to the irradiating unit 12,then the irradiating unit 12 performs control such that the actualvalues of the light emitting intensities are made to be equal to thetarget values.

In addition to the above adjustment, if the luminance feature quantityIh of the region which is within the image Gb captured under thecondition B with the smaller light emitting intensity, among the imagesGa, Gb used for the calculation of the luminance variation dI, and whichcoincides with the in-subject variation region RLH is not larger than athreshold value TLa, then the target value is so adjusted that the lightemitting intensity for the above-mentioned illumination condition B (theillumination condition B with the smaller light emitting intensity) isenlarged. That is, the target value ϕB of the light emitting intensityfor the above-mentioned illumination condition B is enlarged.

The luminance feature amount Ih mentioned here is, for example, amaximum value, a mean value or a median value of the luminance value. Inthe following description, it is assumed that the mean value of theluminance value is used as the luminance feature quantity.

If, for example, the luminance variation dI was calculated using theimage Ga captured at the time point t1 a under an illumination conditionA (light emitting intensity ϕA(t1 a), and the image Gb captured at thetime point t1 b under an illumination condition B (light emittingintensity ϕB(t1 b), then the target value ϕB(t2 b) of the light emittingintensity at the time point t2 b for the next image capturing under theillumination condition B is adjusted according the equation (7) so as tobe enlarged.ϕB(t2b)=ϕB(t1b)+Δϕb  (7)

If the process of the equation (6) and the process of the equation (7)are performed simultaneously, the target value ϕB(t2 b) of the lightemitting intensity after the adjustment is as given by the followingequation (8).ϕB(t2b)=ϕB(t1b)+Δϕa+Δϕb  (8)

Also, if the luminance feature quantity Ih of the region which is withinthe image Ga captured under the condition (A) with the larger lightemitting intensity, among the images Ga, Gb used for the calculation ofthe luminance variation dI, and which coincides with the in-subjectvariation region RLH is larger than a threshold value TLb, then thetarget value ϕA is so adjusted that the light emitting intensity for theabove-mentioned illumination condition A (the condition A with thelarger light emitting intensity) is reduced. That is, the target valueϕA of the light emitting intensity for the above-mentioned illuminationcondition A is reduced.

If, for example, the luminance variation dI was calculated using theimage Ga captured at the time point t1 a under an illumination conditionA (light emitting intensity φA(t1 a)), and the image Gb captured at thetime point t1 b under an illumination condition B (light emittingintensity φB(t1 b)), then the target value φA(t2 a) of the lightemitting intensity at the time point t2 a for the next image capturingunder the illumination condition A is adjusted according to the equation(9) so as to be reduced.φA(t 2 a)=φA(t1a)−Aφc  (9)

If the process of the equation (5) and the process of the equation (9)are performed simultaneously, the target value ϕA(t2 a) of the lightemitting intensity after the adjustment is as given by the followingequation (10).ϕA(t2a)=ϕA(t1a)−Δϕa−Δϕc  (10)

If there was no in-subject variation region RLH, the target value ϕA(t2a) of the light emitting intensity at the time point t2 a for the nextimage capturing under the illumination condition A, and the target valueϕB(t2 b) of the light emitting intensity at the time point t2 b for thenext image capturing under the illumination condition B are respectivelyadjusted according to the equations (11) and (12). By such adjustment,the difference in the light emitting intensity between the twoillumination conditions A and B is enlarged, so that a greater luminancevariation dI is made to occur in the subject.ϕA(t2a)=ϕA(t1a)+Δϕd  (11)ϕB(t2b)=ϕB(t1b)−Δϕd  (12)

By changing the light emitting intensity in the manner described above,a sufficiently large luminance variation can be made to occur in thesubject region.

In adjusting the target values according to the updating rules indicatedby the equations (5) to (12), the amounts of adjustment Δϕa to Δϕd areset to be sufficiently small compared with the light emittingintensities ϕA and ϕB. If the amounts of adjustment Δϕa to Δϕd are toolarge, the light emitting intensities ϕA and ϕB oscillate, which isundesirable. Also, in order to stabilize the light emitting intensitiesϕA and ϕB, the time delay until the updating is important. The amountsof adjustment Δϕa to Δϕd are set taking account of these points.

The target value adjusting unit 243 writes the target values having beenupdated in the manner described above, in the target value memory 244,and also supplies the target value signal Q representing the targetvalues to the imaging/irradiating control unit 11 b.

A processing procedure in the imaging apparatus according the secondembodiment will now be described with reference to the flowcharts ofFIG. 23 and FIG. 24. The processing procedure shown in FIG. 23 and FIG.24 is generally identical to that of the method shown in FIG. 13, but isdifferent in that the step ST1 is replaced with a step ST1 b, and stepsST31 to ST34 are added. In FIG. 23 and FIG. 24, reference charactersidentical to those in FIG. 13 denote identical or corresponding steps.

In the step ST1 b, the imaging/irradiating control unit 11 b receivesthe target value signal Q representing the target values of theillumination condition and the imaging condition from the target valuecalculation unit 24, and outputs the control signal C11 a and thecontrol signal C11 b according to the target value signal Q.

The illumination condition is controlled so as to be changed, taking twoframe periods as one cycle, as in the step ST1 in FIG. 13.

In the step ST31, the in-subject variation region extraction unit 241extracts an in-subject variation region RLH based on the subject regionH detected in the step ST10 and the texture variation dF calculated inthe step ST9.

Next, in the step ST32, the area determination unit 242 determineswhether the area of the in-subject variation region RLH extracted in thestep ST31 is not smaller than the threshold value TAa.

If the area of the in-subject variation region RLH is smaller than thethreshold value TAa, the process in the particular frame ends.

If the area of the in-subject variation region RLH is not smaller thanthe threshold value TAa, the process proceeds to the step ST33.

In the step ST33, the target value adjusting unit 243 reads the targetvalues stored in the target value memory 244, calculates new targetvalues based on the in-subject variation region RLH extracted in thestep ST31, the texture variation dF calculated in the step ST9, and theimage data (the latest image data) stored in the image memory 14, writesthe newly calculated target values in the target memory 244 (e.g. byoverwriting the old target values), and also supplies the target valuesignal Q representing the new target values to the imaging/irradiatingcontrol unit 11 b.

The calculation of the new target values can be performed in the mannerdescribed with reference to the equations (5) to (12).

Next, in the step ST34, the imaging/irradiating control unit 11 bperforms the generation of the control signals C11 a, C11 b (updating oftheir contents) based on the target value signal Q supplied from thetarget value calculation unit 24.

Based on the control signals C11 a, C11 b, the illumination conditionfor the frame of each order (first frame, or second frame) in each cycleconsisting of two frame periods is updated. After the updating, imagecapturing for each frame is performed using the updated illuminationcondition for the frame of the same order.

In the embodiment described above, the target value calculation unit 24calculates the target value of the light emitting intensity of theilluminating light, as the target value of the illumination condition.Alternatively, the target value of the irradiation distribution of theilluminating light may be calculated. Still alternatively, both of thetarget value of the light emitting intensity of the illuminating lightand the target value of the illumination distribution of theilluminating light may be calculated.

In the embodiment described above, the adjustment of the illuminationcondition (determination of the new target value) is performed based onthe luminance feature quantity of the region which is within the imagesGa, Gb used for the calculation of the luminance variation dI and whichcoincides with the in-subject variation region RLH. However, the presentinvention is not limited to this. For instance, the adjustment of theillumination condition (determination of the new target value) may beperformed based on the luminance feature quantity of the entirety of thesubject region in the images Ga, Gb used for the calculation of theluminance variation dI.

As has been described, according to the imaging apparatus of the presentembodiment, the target value of at least one of the illuminationcondition and the imaging condition is calculated based on at least oneof the luminance feature quantity Ih and the texture variation dF in theregion extracted as the subject region, and the imaging/irradiatingcontrol unit 11 b outputs the control signals C11 a, C11 b based on thetarget value signal Q representing the calculated target value, so thatit is possible to obtain the images suitable for the extraction of thesubject region regardless of the change in the ambient light, therebyimproving the accuracy in the extraction of the subject region.

In particular, if the target value of at least one of the irradiationdistribution of the illuminating light and the light emitting intensityof the illuminating light is calculated as the target value of theillumination condition, it is possible to obtain images which aresuitable for the extraction of the subject region regardless of thelocal change of the ambient light, thereby improving the accuracy in theextraction of the subject region.

Also, in the imaging apparatus according to the present embodiment, whenthe area of the region (the in-subject variation region RLH) where thetexture variation dF calculated using the images of differentillumination conditions is larger than the threshold value TFa is notsmaller than the threshold value (TAa), the target value of at least oneof the illumination condition and the imaging condition is calculatedsuch that the texture variation dF between the images of differentillumination conditions is reduced. Also, when the extracted subjectregion does not include any region (in-subject variation region RLH) inwhich the texture variation dF calculated using images of differentillumination conditions is larger than the threshold value TFa, thetarget value of at least one of the illumination condition and theimaging condition is calculated such that the texture variation dFbetween the images of the different illumination conditions is enlarged.Accordingly, it is possible to obtain images which are suitable for theextraction of the subject region regardless of the change of the ambientlight, thereby improving the accuracy in the extraction of the subjectregion.

Furthermore, in the imaging apparatus according to the presentembodiment, when the area of the region (in-subject variation regionRLH) which is within extracted subject region and in which the texturevariation dF calculated using the images of the different illuminationconditions is larger than the threshold value TFa is not smaller thanthe threshold value TAa, and the luminance feature quantity of theabove-mentioned subject region, e.g., the luminance feature quantity Ihof the in-subject variation region RLH, in the image of the illuminationcondition with the smaller light emitting intensity, among the imagesused for the calculation of the luminance variation dI, is not largerthan the threshold value, then the target value related to theillumination condition is calculated such that the light emittingintensity of the above-mentioned illumination condition (illuminationcondition with the smaller light emitting intensity) is enlarged (thatis, the target value of the light emitting intensity is enlarged). Also,when the area of the region (in-subject variation region RLH) which iswithin the extracted subject region and in which the texture variationdF calculated using the images of the different illumination conditionsis larger than the threshold value TFa is not smaller than the thresholdvalue TFa, and the luminance feature quantity of the above-mentionedsubject region, e.g., the luminance feature quantity Ih of thein-subject variation region RLH, in the image of the illuminationcondition with the larger light emitting intensity, among the imagesused for the calculation of the luminance variation dI, is larger thanthe threshold value, then the target value related to the illuminationcondition is calculated such that the light emitting intensity of theabove-mentioned illumination condition (illumination condition with thelarger light emitting intensity) is reduced (that is, the target valueof the light emitting intensity is reduced). Accordingly, it is passableto obtain images suitable for the extraction of the subject regionregardless of the change in the ambient light, thereby improving theaccuracy in the extraction of the subject region.

Third Embodiment

FIG. 25 is a block diagram showing the configuration of an imagingapparatus according to a third embodiment of the present invention. Theimaging apparatus shown in FIG. 25 is generally identical to the imagingapparatus shown in FIG. 21. However, it differs in that it is providedwith an imaging/irradiating control unit 11 c, a luminance variationcalculation unit 21 c, a texture variation calculation unit 22 c, asubject extraction unit 23 c, and a target value calculation unit 24 cin place of the imaging/irradiating control unit 11 b, the luminancevariation calculation unit 21, the texture variation calculation unit22, the subject extraction unit 23, and the target value calculationunit 24 in FIG. 21. The reference characters identical to those in FIG.21 denote identical or corresponding components, and their descriptionis omitted.

In the second embodiment, the texture variation calculation unit 22calculates a single texture variation dF from two images captured underdifferent illumination conditions at different time points. In contrast,in the third embodiment, the texture variation calculation unit 22 ccalculates a first texture variation dF1 from two images captured underthe same illumination condition at different time points, and alsocalculates a second texture variation dF2 from two images captured underdifferent illumination conditions at different time points.

To enable the calculation of the two texture variations dF1, dF2 asmentioned above, the imaging/irradiating control unit 11 c in FIG. 25alters the illumination condition taking three frame periods as oneoperation period (one cycle), and in each operation period, generates anillumination condition for one frame period, and another illuminationcondition B for two frame periods consecutively. That is, in the firstframe in each cycle, image capturing is performed under the firstillumination condition A, and in the second and third frames, imagecapturing is performed under the second illumination condition B. As aresult, the illumination condition differs between the first frame andthe second frame, and the illumination condition is the same between thesecond frame and the third frame.

The luminance variation dI and the second texture variation dF2 arecalculated from the image Ga of the first frame and the image Gb1 of thesecond frame, and the first texture variation dF1 is calculated from theimage Gb1 of the second frame and the image Gb2 of the third frame.

Then, the subject region is extracted based on the luminance variationdI and the first texture variation dF1 calculated in the mannerdescribed above. Furthermore, the in-subject variation region RLH isextracted based on the subject region extracted in this way and thesecond texture variation dF2.

As shown in FIG. 26, the luminance variation calculation unit 21 cincludes a luminance feature quantity calculation unit 211, a luminancefeature quantity memory 212, an illumination condition determinationunit 213 c, a difference calculation unit 215, and a luminance variationmemory 216.

The luminance feature quantity calculation unit 211 calculates theluminance feature quantity Im of the image of each frame read from theimage memory 14, and stores the calculated luminance feature quantity Imin the luminance feature quantity memory 212.

The illumination condition determination unit 213 c compares theluminance feature quantity Im of the latest image (image of the latestframe) calculated by the luminance feature quantity calculation unit211, and the luminance feature quantity Im of the image one frame beforeand the luminance feature quantity Im of the image two frames before,stored in the luminance feature quantity memory 212. These images areimages of three frames consecutive to each other. The illuminationcondition determination unit 213 c compares the luminance featurequantities Im of the images of the three frames, and determines, basedon the results of the comparison, which of the images is the image Ga ofthe illumination condition A, and which of the images are the imagesGb1, Gb2 of the illumination condition B. It also determines the orderof each image in each cycle of the change of the illumination condition.The result GNb of this determination is supplied to the differencecalculation unit 215 and the texture variation calculation unit 22 c.

The difference calculation unit 215 receives the above mentioned resultCNb of the determination from the illumination condition determinationunit 213 c, and recognizes the order of the latest image in each cycleof the illumination condition change. The difference calculation unit215 subtracts, from the luminance value of each pixel in the first imageGa in each cycle of the illumination condition change (image of theillumination condition A), the luminance value of the pixel at the sameposition in the second image Gb1 in each cycle of the illuminationcondition change (image of the illumination condition B), to determinethe luminance difference, and outputs it as the luminance variation dI.

As was also described in connection with the first embodiment, if theresult of the subtraction is negative, the luminance difference istreated as being zero.

The texture variation calculation unit 22 c outputs the first texturevariation dF1 to the subject extraction unit 23 c, and the secondtexture variation dF2 to the target value calculation unit 24 c.

As shown in FIG. 27, the texture variation calculation unit 22 cincludes a CSLBP feature calculation unit 221, a CSLBP feature memory222, a texture feature calculation unit 223, a texture feature memory224, a feature variation processing unit 225 c, and a texture variationmemory 226.

The CSLBP feature calculation unit 221, the CSLBP feature memory 222,and the texture feature calculation unit 223 in FIG. 27 are identical tothose shown in FIG. 4.

The feature variation processing unit 225 c receives the above-mentioneddetermination result CNb from the illumination condition determinationunit 213 c in the luminance variation calculation unit 21 c, torecognize the order of the latest image in each cycle of theillumination condition change. The feature variation processing unit 225c calculates the texture variation between the first image Ga and thesecond image Gb1 in each cycle as the second texture variation dF2. Italso calculates the texture variation between the second image Gb1 andthe third image Gb2 in each cycle as the first texture variation dF1.

The first texture variation dF1 and the second texture variation dF2having been calculated are stored in the texture variation memory 226.The first texture variation dF1 having been stored is later supplied tothe subject extraction unit 23 c. The second texture variation dF2having been stored is later supplied to the target value calculationunit 24 c.

The subject extraction unit 23 c is similar to the subject extractionunit 23 c explained in the first embodiment.

However, it uses the first texture variation dF1, as the texturevariation.

That is, the subject extraction unit 23 c extracts the subject regionbased on the luminance variation dI calculated by the luminancevariation calculation unit 21 c and the first texture variation dF1calculated by the texture variation calculation unit 22 c, and outputsthe extraction result H to the target value calculation unit 24 c.

The process performed by the subject extraction unit 23 c to extract thesubject region based on the luminance variation dI and the texturevariation dF1 is identical to the process performed by the subjectextraction unit 23 to extract the subject region based on the luminancevariation dI and the texture variation dF in the first embodiment.

The target value calculation unit 24 c reads the images captured underthe two illumination conditions from the image memory 14, receives thesecond texture variation dF2 from the texture variation calculation unit22 c, receives the extraction result of the subject region H from thesubject extraction unit 23 c, calculates, based on these inputs, thetarget value related to at least one of the illumination condition andthe imaging condition, and outputs the target value signal Qrepresenting the calculated target value to the imaging/irradiatingcontrol unit 11 c.

The target value calculation unit 24 c is similar to the target valuecalculation unit 24 in the second embodiment,

and the target value signal Q supplied from the target value calculationunit 24 c to the imaging/irradiating control unit 11 c is similar tothat explained in the second embodiment. However, as the texturevariation, the second texture variation dF2 is supplied.

The method of adjusting the target value will now be explained indetail.

The in-subject variation region extraction unit 241 in the target valuecalculation unit 24 c extracts a region RLH which is positioned withinthe subject region H output from the subject extraction unit 23 c, andin which the texture variation is relatively large.

Whether the texture variation is relatively large is determined based onthe texture variation dF2 output from the texture variation calculationunit 22 c.

For instance, the region in which the above-mentioned texture variationdF2 is larger than the threshold value TFa is determined to be anin-subject variation region RLH, and the region other than thein-subject variation region RLH is determined to be a no-variationregion RLL.

In the second embodiment, TFa<TF was said to be desirable. In the thirdembodiment, there is no such restriction.

By performing such processes, it is possible to extract the region(in-subject variation region) RLH which is within the subject region inwhich the first texture variation dF1 is found to be not larger than thethreshold value TF by the subject extraction unit 23 c, and in which thesecond texture variation dF2 is larger than the threshold value TFa(different from the threshold value TF).

The area determination unit 242 determines whether the area of thein-subject variation region RLH, for example, the number of pixelsincluded in the region RLH extracted by the in-subject variation regionextraction unit 241 is not smaller than the threshold value TAa, andoutputs the result of the determination.

When the area determination unit 242 determines that the area of thein-subject variation region RLH is not smaller than the threshold valueTAa, the target value adjusting unit 243 calculates a new target valueresponsive to the result of the determination.

The new target value is so determined that the difference in the lightemitting intensity between the two illumination conditions A and B isreduced. For instance, if the second texture variation dF2 wascalculated using the image captured at the time point t1 a under theillumination condition A (light emitting intensity ϕA(t1 a)), and theimage captured at the time point t1 b under the illumination condition B(light emitting intensity ϕB(t1 b)), then the target value ϕA(t2 a) ofthe light emitting intensity at the time point t2 a for the next imagecapturing under the illumination condition A, and the target value ϕB(t2b) of the light emitting intensity at the time point t2 b for the nextimage capturing under the illumination condition B are respectivelyadjusted according to the equations (13) and (14). By adjusting thetarget values in this way, the difference between the target values ofthe light emitting intensities for the two illumination conditions A andB is reduced.ϕA(t2a)=ϕA(t1a)−Δϕa  (13)ϕB(t2b)=ϕB(t1b)+Δϕa  (14)

In the equations (13) and (14), Δϕa denotes an amount of adjustment.

The target values of the light emitting intensities are thus adjustedand the adjusted target values are supplied to the irradiating unit 12,then the irradiating unit 12 performs control such that the actualvalues of the light emitting intensities are made to be equal to thetarget values.

In addition to the above-mentioned adjustment, if the luminance featurequantity Ih of the region which is within the image Gb1 captured underthe condition B with the smaller light emitting intensity, among theimages Ga, Gb1 used for the calculation of the luminance variation dI,and which coincides with the in-subject variation region RLH, is notlarger than the threshold value TLa, then the target value is adjustedso that the light emitting intensity of the above-mentioned illuminationcondition B (the illumination condition B with the smaller lightemitting intensity) is enlarged. That is, the target value φB of thelight emitting intensity of the illumination condition B is enlarged.

The luminance feature amount Ih mentioned here is, for example, amaximum value, a mean value or a median value of the luminance value. Inthe following description, it is assumed that the mean value of theluminance value is used as the luminance feature quantity.

If, for example, the luminance variation dI was calculated using theimage Ga captured at the time point t1 a under an illumination conditionA (light emitting intensity ϕA(t1 a), and the image Gb captured at thetime point t1 b under an illumination condition B (light emittingintensity ϕB(t1 b), then the target value ϕB(t2 b) of the light emittingintensity at the time point t2 b for the next image capturing under theillumination condition B is adjusted according the equation (15) so asto be enlarged.ϕB(t2b)=ϕB(t1b)+Δϕb  (15)

If the process of the equation (14) and the process of the equation (15)are performed simultaneously, the target value ϕB(t2 b) of the lightemitting intensity after the adjustment is as given by the followingequation (16).ϕB(t2b)=ϕB(t1b)+Δϕa+Δϕb  (16)

Also, if the luminance feature quantity Ih of the region which is withinthe image Ga captured under the condition (A) with the larger lightemitting intensity, among the images Ga, Gb1 used for the calculation ofthe luminance variation dI, and which coincides with the in-subjectvariation region RLH is larger than the threshold value TLb, then thetarget value ϕA is so adjusted that the light emitting intensity for theabove-mentioned illumination condition A (the condition A with thelarger light emitting intensity) is reduced. That is, the target valueϕA of the light emitting intensity for the above-mentioned illuminationcondition A is reduced.

If, for example, the luminance variation dI was calculated using theimage Ga captured at the time point t1 a under an illumination conditionA (light emitting intensity φA(t1 a)), and the image Gb1 captured at thetime point t1 b under an illumination condition B (light emittingintensity φB(t1 b)), then the target value φA(t2 a) of the lightemitting intensity at the time point t2 a for the next image capturingunder the illumination condition A is adjusted according to the equation(17) so as to be reduced.φA(t2a)=φA(t1a)−Δφc  (17)

If the process of the equation (13) and the process of the equation (17)are performed simultaneously, the target value ϕA(t2 a) of the lightemitting intensity after the adjustment is as given by the followingequation (18).ϕA(t2a)=ϕA(t1a)+Δϕa−Δϕc  (18)

If there was no in-subject variation region RLH, the target value ϕA(t2a) of the light emitting intensity at the time point t2 a for the nextimage capturing under the illumination condition A, and the target valueϕB(t2 b) of the light emitting intensity at the time point t2 b for thenext image capturing under the illumination condition B are respectivelyadjusted according to the equations (19) and (20). By such adjustment,the difference in the light emitting intensity between the twoillumination conditions A and B is enlarged, so that a greater luminancevariation dI is made to occur in the subject.ϕA(t2a)=ϕA(t1a)+Aϕd  (19)ϕB(t2b)=ϕB(t1b)−Δϕd  (20)

By changing the light emitting intensity in the manner described above,a sufficiently large luminance variation can be made to occur in thesubject region.

In adjusting the target values according to the updating rules indicatedby the equations (13) to (20), the amounts of adjustment Δϕa to Δϕb areset to be sufficiently small compared with the light emittingintensities ϕA and ϕB. If the amounts of adjustment Δϕa to Δϕd are toolarge, the light emitting intensities ϕA and ϕB oscillate, which isundesirable. Also, in order to stabilize the light emitting intensitiesϕA and ϕB, the time delay until the updating is important. The amountsof adjustment Δϕa to Δϕd are set taking account of these points.

The target value adjusting unit 243 writes the target values having beenupdated in the manner described above, in the target value memory 244,and also supplies the target value signal Q representing the targetvalues to the imaging/irradiating control unit 11 c.

A processing procedure in the imaging apparatus according the thirdembodiment will now be described with reference to the flowcharts ofFIG. 28 and FIG. 29. The processing procedure shown in FIG. 28 and FIG.29 is generally identical to that shown in FIG. 23 and FIG. 24, but isdifferent in that the steps ST1 b, ST7, ST8, ST9, ST10, and ST31 arereplaced with steps ST1 c, ST7 c, ST8 c, ST9 c, ST10 d, and ST31 d. InFIG. 28 and FIG. 29, reference characters identical to those in FIG. 23and FIG. 24 denote identical or corresponding steps.

In the step ST1 c, the imaging/irradiating control unit 11 c receivesthe target value signal Q representing the target values of theillumination condition and the imaging condition from the target valuecalculation unit 24 c, and outputs the control signal C11 a and thecontrol signal C11 b according to the target value signal Q.

The step ST1 c is similar to the step ST1 b in FIG. 23, but differs inthat, in the step ST1 c in FIG. 28, the illumination condition ischanged taking three frame periods as one cycle.

In the step ST7 c, a determination is made on the illuminationconditions of the images, for the combination of the latest image andthe image one frame before. In this determination, whether thecombination of the latest image and the image one frame before is acombination (AB) of an image of a first frame and a second frame in athree-frame operation periods, or a combination (BB) of an image of asecond frame and a third frame, or a combination (BA) of an image of athird frame, and a first frame (in the next three-frame operationperiod). This determination is performed by storing and comparing thevariations in the brightness values of the images of the frames suppliedone after the other, as was explained in connection with the firstembodiment.

Alternatively, the control signals C11 a, C11 b may be obtained from theimaging/irradiating control unit 11 c, and the determination may be madebased on the control signals C11 a, C11 b.

If, in the step ST7 c, the combination is found to be one (AB) of thefirst frame image and the second frame image, the process proceeds tothe step ST8 c and the step ST9 c.

The processes in the step ST8 c and the step ST9 c are similar theprocesses in the step ST8 and the step ST9 in the first embodiment.

That is, in the step ST8 c, the difference calculation unit 215 in theluminance variation calculation unit 21 c calculates the luminancevariation dI of each pixel between the image of the latest frame (secondframe image) and the image one frame before (first frame image). Theluminance variation dI of each pixel that has been calculated is storedin the luminance variation memory 216.

In the step ST9 c, the texture variation calculation unit 22 ccalculates the texture variation of each pixel between the image of thelatest frame (second frame image) and the image one frame before (firstframe image), and outputs the calculated texture variation as the secondtexture variation dF2, and causes it to be stored in the texturevariation memory 226.

After the steps ST8 c and ST9 c, the process in this frame period ends.

The luminance variation dI stored in the luminance variation memory 216in the step ST8 c, and the second texture variation dF2 stored in thetexture variation memory 226 in the step ST9 c are used in the nextframe period.

If, in the step ST7 c, the combination is one (BB) of the second frameimage and the third frame image, the process proceeds to a step ST9 d.

In the step ST9 d, the texture variation calculation unit 22 ccalculates the texture variation of each pixel between the image of thelatest frame (third frame image) and the image one frame before (secondframe image), and outputs the calculated texture variation as the firsttexture variation dF1, and causes it to be stored in the texturevariation memory 226.

In the step ST10 d, following the step ST9 d, the subject region isextracted. This process is identical to the step ST10 in FIG. 13.However, while the texture variation dF is used in the step ST10 in FIG.13, the first texture variation dF1 calculated in the step ST9 d is usedin the step ST10 d in FIG. 29. Also, the luminance variation dI that wascalculated in the step ST8 c one frame period before is used.

After the step ST10 d, the process proceeds to the step ST31 d.

The process in the step ST31 d is similar to the process in the stepST31 in FIG. 24, but the second texture variation dF2 calculated in thestep ST9 c is used for the detection of the in-subject variation regionRLH.

That is, in the step ST31 d, the in-subject variation region extractionunit 241 extracts the in-subject variation region RLH based on thesubject region H detected in the step ST10 d, and the second texturevariation dF2 calculated in the step ST9 c.

After the step ST31 d, the processes of the steps ST32, ST33, and ST33are performed. These processes are similar to the steps ST32, ST33, andST33 in FIG. 24.

That is, in the step ST32, the area determination unit 242 determineswhether the area of the in-subject variation region RLH extracted in thestep ST31 d is not smaller than the threshold value TAa.

If the area of the in-subject variation region RLH is smaller than thethreshold value TAa, the process in the particular frame ends.

If the area of the in-subject variation region RLH is not smaller thanthe threshold value TAa, the process proceeds to the step ST33.

In the step ST33, the target value adjusting unit 243 reads the targetvalues stored in the target value memory 244, calculates new targetvalues based on the in-subject variation region RLH extracted in thestep ST31 d, the second texture variation dF2 calculated in the step ST9c (in the preceding frame period), and the image data (image data of thelatest frame) stored in the image memory 14, writes the newly calculatedtarget values in the target value memory 244 (e.g. by overwriting theold target values), and also supplies the target value signal Qrepresenting the new target values to the imaging/irradiating controlunit 11 c.

The calculation of the new target values can be performed in the mannerdescribed with reference to the equations (13) to (20).

Next, in the step ST34, the imaging/irradiating control unit 11 cperforms the generation of the control signals C11 a, C11 b (updating oftheir contents) based on the target value signal Q supplied from thetarget value calculation unit 24 c.

Based on the control signals C11 a, C11 b, the illumination conditionfor the frame of each order (first frame, second frame, or third frame)in each cycle consisting of three frame periods is updated. After theupdating, image capturing for a frame of each order is performed usingthe illumination condition updated for the frame of the same order.

In the third embodiment, the process of updating the target values,which, in the second embodiment, is performed using the texturevariation dF, is performed using the second texture variation dF2.

The second texture variation dF2 is a texture variation between theimages obtained by image capturing at different time points and underdifferent illumination conditions, so that it is identical to the“texture variation” in the second embodiment. Accordingly, effectssimilar to those of the second embodiment can be obtained by the thirdembodiment.

In the second and third embodiments, the target value adjusted by thetarget value calculation unit 24 or 24 c is the target value of thelight emitting intensity of the illuminating light. However, the presentinvention is not limited to this. The above-mentioned target value maybe a target value of at least one of the shape of the irradiationdistribution of the illuminating light, the light emitting time of theilluminating light, the exposure time of the imaging unit 13, theaperture of the imaging unit 13 and the gain of the imaging unit 13. Forinstance, when the in-subject variation region RLH or the no-variationregion RLL is positioned locally in an image, the irradiationdistribution may be adjusted to locally adjust the light emittingintensity. Also, when the luminance feature quantity Ih in thein-subject variation region RLH is not larger than a threshold valueTLa′, or larger than a threshold value TLb′, the exposure time may beadjusted to adjust the luminance feature quantity Ih. The thresholdvalues TLa′ and TLb′ used in this case may be identical to or differentfrom the threshold values TLa and TLb used for the above-mentioneddecision for the adjustment of the light emitting intensity.

Also, in the second and third embodiments, the calculations of theluminance variation and the texture variation are performed between twoimages. However, it may be so arranged that the calculations of theluminance variation and the texture variation are performed betweenthree or more images. In summary, it is satisfactory if the calculationsof the luminance variation and the texture variation are performedbetween a plurality of images. When the calculation of the luminancevariation is performed between three or more images, images capturedunder mutually different illumination conditions are used. When thecalculation of the texture variation is performed between three or moreimages, images captured at mutually different time points are used. Theimages captured at mutually different time points may be images capturedunder mutually different illumination conditions. Accordingly, they maybe the same as the images which are used for the calculation of theluminance variation.

When the target value of the illumination condition or the imagingcondition is, adjusted from the luminance variation and the texturevariation calculated using three or more images, as the process usingthe “image of the illumination condition with the smaller light emittingintensity” in the above description, a process using the “image of theillumination condition with the smallest light emitting intensity” maybe performed; as the process using the “image of the illuminationcondition with, the larger light emitting intensity” in the abovedescription, a process using the “image of the illumination conditionwith the largest light emitting intensity” may be performed; as theprocess on (adjustment of the target value of) the “smaller lightemitting intensity” in the above description, a process on the smallestlight emitting intensity” may be performed; and, as the process on(adjustment of the target value of) the “larger light emittingintensity” in the above description, a process on the largest lightemitting intensity” may be performed.

In the second and third embodiments, the magnitude of threshold valueTAa was a fixed value regardless of the area of the in-subject variationregion RLH. However, the threshold value may be adaptively changed. Forinstance, the magnitude of the threshold value TAa may be modifieddepending on the area of the extracted subject region. In this case, ifthe area of the extracted subject region is large, the threshold valueTAa may be adjusted to be enlarged, and if the area of the extractedsubject region is small, the threshold value TAa may be adjusted to bereduced.

Also, in the second and third embodiments, the magnitudes of the amountsof adjustment Δϕa to Δϕd are fixed values.

They may however be changed adaptively. For instance, the magnitudes ofthe amounts of adjustment Δϕa to Δϕd may bemodified according to theproportion between the areas of the in-subject variation region RLH andthe no-variation region RLL in the subject region.

Also, the amounts of adjustment Δϕa to Δϕd for all the illuminationconditions need not be of the same magnitude, as shown in the equations(5) to (20), but the amounts of adjustment Δϕa to Δϕd of differentmagnitudes may be used.

Furthermore, in the above example, the luminance variation calculationunit 21 c notifies the texture variation calculation unit 22 c of theorder of each of the three images read from the image memory 14.However, the texture variation calculation unit 22 c may make thedetermination for itself. In this case, the determination may be madefrom the luminance feature quantity Im of the images, as in theluminance variation calculation unit 21 c.

Fourth Embodiment

FIG. 30 is a block diagram showing the configuration of an operationapparatus according to a fourth embodiment of the present invention. Theoperation apparatus shown in FIG. 30 is a combination of an imagingapparatus similar to the imaging apparatus shown in FIG. 21, and asubject recognition unit 25 and an operation determination unit 26.However, the texture variation calculation unit 22 in FIG. 21 isreplaced with a texture variation calculation unit 22 d. Referencecharacters identical to those in FIG. 21 denote identical orcorresponding components, and their description is omitted.

Also, in the present embodiment, the subject is assumed to be a hand ofa person.

The texture variation calculation unit 22 d successively reads aplurality of images stored in the image memory 14, determines a texturefeature F for each of the pixels constituting each image, calculates atexture variation dF between two images captured at two different timepoints, supplies the calculated texture variation dF to the subjectextraction unit 23, the target value calculation unit 24 and the subjectrecognition unit 25, and also supplies the texture feature F to thesubject recognition unit 25.

As shown in FIG. 31, the texture variation calculation unit 22 dincludes a CSLBP feature calculation unit 221, a CSLBP feature memory222, a texture feature calculation unit 223, a texture feature memory224 d, a feature variation processing unit 225 and a texture variationmemory 226.

Among the components shown in FIG. 31, the CSLBP feature calculationunit 221, the CSLBP feature memory 222, and the feature variationprocessing unit 225 are identical to those shown in FIG. 4.

The texture feature memory 224 d has functions similar to those of thetexture feature memory 224 in FIG. 4, but differs from the texturefeature memory 224 in FIG. 4, in that the data stored in it can also beread by a component other than the feature variation processing unit225, in particular by the subject recognition unit 25.

The subject recognition unit 25 receives the extraction result H of thesubject region H supplied from the subject extraction unit 23, thetexture variations dF and the texture features F supplied from thetexture variation calculation unit 22 d, and the images G read from theimage memory 14, recognizes at least one of the shape and the motion ofthe subject based on these inputs, and outputs the result of therecognition to the operation determination unit 26.

The subject recognition unit 25 recognizes a gesture performed, forexample, by a person. The gesture, may be a shape gesture by the shapeof the hand (‘rock’, ‘scissors’, ‘paper’ etc.), or a motion gesture bythe motion of the hand (lateral movement, longitudinal movement,twisting movement).

As shown in FIG. 32, the subject recognition unit 25 includes a shaperecognition unit 251, a motion recognition unit 252, and a referencedata memory 253.

With regard to the shape gesture, reference data Ss representing apredetermined feature quantity of the shape is registered in advance inthe reference data memory 253, and the shape recognition unit 251recognizes the shape gesture by comparing the shape of the subjectcaptured by the imaging unit 13, with the reference data Ss.

The reference data Ss represents the conditions to be satisfied in orderfor the shape of the hand to be recognized as intended for a particulargesture. The determination as to whether the conditions are satisfiedincludes a threshold processing on the feature quantity of the shape ofthe hand (determination as to whether the feature quantity of the shapeof the hand has a predetermined relation with the threshold value).

For instance, the determination is made based on whether the differencefrom the data which represents the shape of the hand intended for aparticular gesture and which is acquired in advance is not larger than apredetermined threshold value, and in such a case, the data having beenacquired and the data representing the threshold value in combinationconstitute the reference data Ss.

The reference data Ss can be modified by learning. Known methods of thelearning include, for example, a method called AdaBoost, and a methodcalled Support Vector Machine.

As the feature quantity of the shape of the hand, the feature (quantityof the boundary lines between the hand and the background (edge of thehand) may be used. Alternatively, the texture features of the handregion (subject region) may be used.

As the boundary line between the hand and the background, the datarepresenting the countour of the subject region extracted by the subjectextraction unit 23 may be used. In this case, the shape recognition unit251 identifies the shape of the hand based on the data representing theextraction result of the subject region H from the subject extractionunit 23.

It is also possible to extract the boundary line between the hand andthe background using the texture features F.

For instance, by analyzing the boundary line between the hand and thebackground, it is possible to determine the shape of the hand, e.g.,whether the hand is in the clenched state (rock) or open state (paper).For the extraction of the boundary line between the hand and thebackground, the difference in the texture feature corresponding todifferences in the patterns, the unevenness, or the reflectivity,between the hand and the background may be used. In this case, the shaperecognition unit 251 identifies the shape of the hand based on thetexture features F stored in the texture feature memory 224 d.

Also, within the hand region, a difference in the unevenness of thesurface may appear depending on the shape of the fingers or the like.For instance, there is a difference in the unevenness of the surfacebetween a situation in which the hand is clenched (rock state) and thepalm is hidden by fingers, and a situation in which the hand is opened(paper state) and the palm is visible. It is possible to makedistinction based on such a difference using the texture feature.Accordingly, by using the texture features, it is possible to extractthe feature representing the particular shape of the hand (rock, paper,or the like). In this case, the shape recognition unit 251 identifiesthe shape of the hand based on the texture features F pertaining to thepixels in the subject region H extracted by the subject extraction unit23, among the texture features F stored in the texture feature memory224 d.

With regard to the motion gesture, reference data Sm representing apredetermined feature quantity of the motion is registered in advance inthe reference data memory 253, and the motion recognition unit 252recognizes the motion gesture by comparing the motion of the subjectcaptured by the imaging unit 13 with the reference data Sm.

The reference data Sm represents the conditions to be satisfied in orderfor the motion of the hand to be recognized as intended for a particulargesture. The determination as to whether the conditions are satisfiedincludes a threshold processing on the feature quantity of the motion ofthe hand (determination as to whether the feature quantity of the motionof the hand has a predetermined relation with the threshold value).

For instance, the determination is made based on whether the differencefrom the data which represents the motion of the hand intended for aparticular gesture and which is acquired in advance is not larger than apredetermined threshold value, and in such a case, the data having beenacquired and the data representing the threshold value in combinationconstitute the reference data Sm.

As was also described in connection with the gesture by the shape of thehand, the reference data Sm for the gesture by the motion of the handcan also be modified by learning.

As the feature quantity of the motion, the change in the position, thevelocity, and the acceleration of the center of the subject may be used.The texture variations dF may also be used.

With regard to the lateral movement and the longitudinal movement, it ispossible to observe the change in the position, the velocity, theacceleration or the like by tracking the center position of the subject,and they may be used as the feature quantity. For the identification ofthe central position, it is necessary to identify the shape of the hand,in particular the boundary line between the hand and the background.

For the identification of the shape of the hand, data representing theextraction result of the subject region H from the subject extractionunit 23 may be used. In this case, the motion recognition unit 252identifies the shape of the hand based on the texture features F storedin the texture feature memory 224 d.

For the identification of the shape of the hand, it is possible to usethe texture features F stored in the texture feature memory 224 d, aswas also described in connection the shape recognition unit 251. In thiscase, the motion recognition unit 252 identifies the shape of the handbased on the texture features F stored in the texture feature memory 224d.

For the identification of the twisting movement or the like with Whichthe change in the boundary line between the hand and the background issmall, a time series of the texture variations dF or the texturefeatures F output from the texture variation calculation unit 22 d maybe used as the feature quantity. In this case, the motion recognitionunit 252 identifies the motion of the hand based on the texturevariations dF pertaining to the pixels in the subject region H extractedby the subject extraction unit 23, among the texture variations dFstored in the texture variation memory 226, or the texture features Fpertaining to the pixels in the subject region H extracted by thesubject extraction unit 23, among the texture features F stored in thetexture feature memory 224 d.

It is also possible to use the luminance variations dI calculated by theluminance variation calculation unit 21, for the purpose of thecalculation of the feature quantity of the motion. For instance, it ispossible to identify the motion of the hand toward or away from thecamera, by using the time series of the mean values of the luminancevariations dI in the subject region H, as the feature quantity of themotion. For instance, if the mean value of the luminance variations dIin the subject region H is increased with time, the hand can bedetermined to be moving toward the camera. If the mean value of theluminance variations dI is decreased with time, the hand is determinedto be moving away from the camera.

Because the images suitable for the calculation of the texture featuresF are obtained by adjusting the illumination conditions using the targetvalues calculated by the target value calculation unit 24, the texturefeatures F can be calculated stably regardless of the ambient light, andbecause the texture features F are used for the calculation of thefeature quantity of the shape or motion, the gesture recognition can beperformed with a high accuracy.

The operation determination unit 26 generates a command for operating adevice, based on the recognition result Rs output from the shaperecognition unit 251 in the subject recognition unit 25, and therecognition result Rm, output from the motion recognition unit 252, andoutputs the command. For instance, device operation commands are presetfor the types of gestures to be recognized by the subject recognitionunit 25, and the command is generated based on the recognition results,and output.

For a case in which the operated device is a vehicle-mounted device,FIG. 33 shows an exemplary correspondence between the types of thegestures and the operation commands for operating the vehicle-mounteddevice. For example, when a gesture of “rock” is recognized, a commandis generated and output to switch the display screen to a map guidancescreen. When a gesture of “scissors” is recognized, a command isgenerated and output to switch the display screen to an audio screen.When a gesture of “paper” is recognized, a command is generated andoutput to switch the display screen to an air conditioner adjustmentscreen.

The processes for the extraction of the subject in the subjectextraction unit 23 in the operation apparatus of the fourth embodimentare identical to those shown in FIG. 23 and FIG. 24.

The operation apparatus of the fourth embodiment performs the processesof the gesture determination, and the device operation based on thegesture determination, using the result of the calculation of thetexture features F in the step ST6 in FIG. 23, and the result of thesubject extraction in the step ST10 in FIG. 24.

The processes of the gesture determination and the device operationbased on the determination result will now be described with referenceto FIG. 34.

In a step ST41, the subject recognition unit 25 receives the result ofthe subject region extraction supplied from the subject extraction unit23, the texture variations dF and the texture features F supplied fromthe texture variation calculation unit 22 d, and the images G read fromthe image memory 14, calculates the feature quantity of at least one ofthe shape and motion of the subject, compares the calculated featurequantity with the reference data (representing the feature quantity ofthe standard shape or the standard motion of the subject) registered inadvance in the reference data memory 253, and outputs at least one ofthe shape and motion of the subject as the recognition result to theoperation determination unit 26.

In a step ST42, a command for operating the device is generated andoutput based on the recognition result from the subject recognition unit25.

As has been described, according to the operation apparatus of thepresent embodiment, the extraction result of the subject region which isstable against the changes in the ambient light is used, so that it ispossible to obtain the subject recognition result which is stableagainst the ambient light.

Also, since the recognition of the shape of the subject is performedbased on the texture features used for the subject region extraction, itis possible to obtain the subject recognition result which is stableagainst the ambient light.

Furthermore, since the recognition of the motion of the subject isperformed based on the texture features used for the subject regionextraction, it is possible to obtain the subject recognition resultwhich is stable against the ambient light.

Furthermore, since the subject recognition result which is stableagainst the changes in the ambient light is used, it is possible toperform the device operation using at least one of the shape and motionof the subject, even in a situation in which the ambient light ischanged.

In the present embodiment, description was made of a case in which theoperated device is a vehicle-mounted device, as an example of the deviceoperation by means of the operation determination unit. However, thepresent invention is not limited to this. The present invention isapplicable to cases in which a video device such as a television set, asignage or the like, or a wearable device of a glass type, a bracelettype, a finger ring type, a garment type, or the like is operated.

Furthermore, in the present embodiment, description was made on a casein which the gesture of the subject is recognized, and the operation isperformed based on the result of the recognition. The present inventionhowever is not limited to this scheme. That is, the present invention isapplicable to a case in which any action or state, other than thegesture, of a person is recognized, and the operation is performed basedon the result of the recognition. For instance, the present invention isapplicable to a case in which the orientation of the face or theopening/closing of an eye or mouth, or a looking-off state, a dozingstate, a state in which the person is talking, a state in which theperson is dozy, a state in which the person is tired, or a state inwhich the attentiveness is lowered, accompanied with any of theorientation of the face or the opening/closing of an eye or mouth, isrecognized, and the operation is performed based on the result of therecognition. The operation based on the result of the recognition may bea process of switching the operation of the device, an operation ofbraking a vehicle, generation of an alarming sound.

Description has been made of an operation apparatus provided with theimaging apparatus of the second embodiment. It is also possible toconfigure an operation apparatus provided with the imaging apparatus ofthe first embodiment or the imaging apparatus of the third embodiment,in place of the imaging apparatus of the second embodiment.

So far the present invention has been described as the imaging apparatusand the operation apparatus. However, an imaging method implemented byusing the above-mentioned imaging apparatus and an operating methodimplemented by using the above-mentioned operation apparatus are alsopart of the present invention.

Part of the processes performed in the above imaging apparatus or theoperation apparatus, or part of the processes performed in the aboveimaging method or the operating method may be executed by a computerincluding a processor. Accordingly, a program for causing a computer toexecute the processes performed in the above imaging apparatus or theoperation apparatus, or the processes performed in the above imagingmethod or the operation method, and a computer-readable recording mediumin which such a program is recorded are also part of the presentinvention.

FIG. 35 shows an example of the computer together with the irradiatingunit 12 and the imaging unit 13. The irradiating unit 12 and the imagingunit 13 have functions similar to the irradiating unit 12 and theimaging unit 13 in, for example, of FIG. 1.

The computer in FIG. 35 includes a processor 51, a program memory 52, adata memory 53, a data input interface 54, a control output interface55, and a data output interface 56, which are interconnected by a databus 57.

The processor 51 operates in accordance with the program stored in theprogram memory 52, and performs the processes similar to the processesperformed by the imaging/irradiating control unit 11, the luminancevariation calculation unit 21, the texture variation calculation unit22, and the subject extraction unit 23 in FIG. 1.

The control output interface 55 is connected to the irradiating unit 12and the imaging unit 13, and supplies the control signal C11 a from theprocessor 51 to the irradiating unit 12, and supplies the control signalC11 b to the imaging unit 13.

The data input interface 54 is connected to the imaging unit 13, andreceives the captured image output from the imaging unit 13.

The processor 51 performs, for example, the processes similar to theprocesses performed by the luminance variation calculation unit 21, thetexture variation calculation unit 22, and the subject extraction unit23 in the first embodiment, on the captured image input via the datainput interface 54, and outputs the result of the subject extraction viathe data output interface 56.

The data memory 53 has a role similar to that of the image memory 14 inFIG. 1.

The data memory 53 may also have the roles of the luminance featurequantity memory 212 and the luminance variation memory 216 (FIG. 3) inthe luminance variation calculation unit 21 in FIG. 1, and the texturefeature memory 224 and the texture variation memory 226 (FIG. 4) in thetexture variation calculation unit 22.

A processing procedure by the processor 51 is similar to that describedwith reference to FIG. 13 and FIG. 14 in connection with the firstembodiment.

A plurality of computers, each as shown in FIG. 35, may be provided andmay be made to perform the processes of the respective parts of theimaging apparatus. This also applies when the processes in the imagingmethod is performed by computers.

Description has been made of a case in which the processes in theimaging apparatus of the first embodiment are performed by a computer.Similar description applies when the processes in the imaging apparatusof the second or third embodiment, or the processes in the imagingmethod implemented by the imaging apparatus, or the processes in theoperation apparatus of the fourth embodiment, or the operation methodimplemented in the operation apparatus are performed by a computer.

DESCRIPTION OF THE REFERENCE CHARACTERS

11,11 b,11 c: imaging/irradiating control unit; 12: irradiating unit;13: imaging unit; 14: image memory; 21, 21 c: luminance variationcalculation unit; 22,22 c,22 d: texture variation calculation unit;23,23 c: subject extraction unit; 24,24 c: target value calculationunit; 25: subject recognition unit; 26: operation determination unit;51: processor; 52: program memory; 53: data memory; 54: data inputinterface; 55: control output interface; 56: data output interface; 121:LED; 122: light emission control unit; 211 luminance feature quantitycalculation unit; 212: luminance feature quantity memory; 213,213 c:illumination condition determination unit; 215: difference calculationunit; 216: luminance variation memory; 221 CSLBP feature calculationunit; 222: CSLBP feature memory; 223, 223 a: texture feature calculationunit; 224, 224 d: texture feature memory; 225, 225 a, 225 c: featurevariation processing unit; 226: texture variation memory; 231: luminancevariation comparison unit; 232: texture variation comparison unit; 233:region determination unit; 234: gate unit; 241: in-subject variationregion extraction unit; 242: area determination unit; 243: target valueadjusting unit; 244: target value memory; 251: shape recognition unit;252: motion recognition unit; 253: reference data memory; 2211: regiondividing unit; 2212-1 to 2212-16: CSLBP feature reading unit; 2213-1 to2213-16: histogram generating unit; 2214: concatenating unit; 2215:normalizing unit; 2216: clipping unit; H: subject; B1: backgroundelement; B2: background part.

What is claimed is:
 1. An imaging apparatus comprising: one or moreprocessors; a memory storing a program that when executed by the one ormore processors causes one or more processors to function as animaging/irradiating control unit for generating an illuminationcondition control signal for controlling an illumination condition, andan imaging condition control signal for controlling an imagingcondition; said illumination condition control signal causing a subjectto be illuminated with a plurality of mutually different illuminationconditions; said imaging condition control signal causing a subjectimage to be captured with an imaging condition to generate capturedimages; a luminance variation calculation unit for calculating, using aplurality of captured images obtained by capturing images underdifferent illumination conditions, a luminance variation pertaining toeach pixel between the plurality of captured images; a texture variationcalculation unit for calculating, using a plurality of captured imagesobtained by capturing images at different time points, a texturevariation pertaining to each pixel between the plurality of capturedimages; and a subject extraction unit for extracting a subject regionbased on the luminance variation and the texture variation, wherein saidtexture variation calculation unit calculates, as the texture variation,a first texture variation using a plurality of captured images obtainedby capturing images under identical illumination conditions at differenttime points, and a second texture variation using the plurality ofcaptured images which are used for the calculation of the luminancevariation.
 2. The imaging apparatus as set forth in claim 1, whereinsaid texture variation calculation unit calculates the second texturevariation using a plurality of captured images obtained by the imagecapturing under mutually different illumination conditions at differenttime points.
 3. The imaging apparatus as set forth in claim 1, whereinsaid texture variation calculation unit calculates texture features foreach of the plurality of captured images, and calculates a variation inthe texture feature between the plurality of captured images as thetexture variation.
 4. The imaging apparatus as set forth in claim 3,wherein said subject extraction unit extracts, from a captured imageused for at least one of the calculation of the luminance variation andthe calculation of the texture variation, or a luminance differenceimage constituted of the luminance variations of the respective pixels,a region in which the luminance variation is larger than a firstthreshold value, and the texture variation is not larger than a secondthreshold value, as the subject region.
 5. An operation apparatuscomprising: the imaging apparatus as set forth in claim 3; and whereinthe one or more processors are further caused to function as a subjectrecognition unit for calculating at least one of a feature quantity of ashape or a feature quantity of motion of the subject based on the resultof the extraction of the subject region by said subject extraction unit,and comparing the calculated feature quantity with reference datarepresenting a predetermined feature quantity of the shape or motion, torecognize the shape or the motion of the subject, wherein said subjectrecognition unit calculates the feature quantity of the shape based onthe texture feature calculated by said texture variation calculationunit.
 6. The imaging apparatus as set forth in claim 1, wherein theplurality of illumination conditions differ in at least one of anirradiation distribution of the illuminating light and a light emittingintensity of the illuminating light.
 7. The imaging apparatus as setforth in claim 1, wherein the plurality of illumination conditionsinclude a first illumination condition in which the illuminating lightis irradiated with a first light emitting intensity, and a secondillumination condition in which the illuminating light is irradiatedwith a second light emitting intensity which is smaller than the firstlight emitting intensity.
 8. The imaging apparatus as set forth in claim1, wherein the one or more processors are further caused to function as:a target value calculation unit for determining a target value of theimaging condition, based on at least one of a luminance feature quantityand the texture variation of the captured image in the extracted subjectregion, wherein said imaging/irradiating control unit generates theimaging condition control signal based on the target value of theimaging condition.
 9. The imaging apparatus as set forth in claim 1,wherein the one or more processors are further caused to function as: atarget value calculation unit for determining a target value of theillumination condition based on at least one of a luminance featurequantity and the texture variation of the captured image in theextracted subject region, wherein said imaging/irradiating control unitgenerates the illumination condition control signal based on the targetvalue of the illumination condition.
 10. The imaging apparatus as setforth in claim 9, wherein when the luminance feature quantity of theextracted subject region, in the captured image obtained by the imagecapturing under the illumination condition with the smallest lightemitting intensity, among the plurality of captured images used f©r thecalculation of the luminance variation, is not larger than a fifththreshold value, said target value calculation unit enlarges the targetvalue of the light emitting intensity used for the illuminationcondition with the smallest light emitting intensity, and when theluminance feature quantity of the extracted subject region, in thecaptured image obtained by the image capturing under the illuminationcondition with the largest light emitting intensity, among the pluralityof captured images used for the calculation of the luminance variation,is larger than a sixth threshold value, said target value calculationunit reduces the target value of the light emitting intensity used forthe illumination condition with the largest light emitting intensity.11. The imaging apparatus as set forth in claim 9, wherein the targetvalue of the illumination condition is a target value of at least one ofan irradiation distribution of the illuminating light, and a lightemitting intensity of the illuminating light.
 12. The imaging apparatusas set forth in claim 9, wherein said subject extraction unit performsthe extraction of the subject region based on the luminance variationand the first texture variation; when an area of a part of the extractedsubject region in which the second texture variation is larger than athird threshold value is not smaller than a fourth threshold value, saidtarget value calculation unit adjusts the target value of theillumination condition such that a difference in the light emittingintensity between the plurality of illumination conditions is reduced,and when the second texture variation is not larger than the thirdthreshold value throughout the extracted subject region, said targetvalue calculation unit adjusts the target value of the illuminationcondition such that the difference in the light emitting intensitybetween the plurality of illumination conditions is enlarged.
 13. Anoperation apparatus comprising: the imaging apparatus as set forth inclaim 1; and wherein the one or more processors are further caused tofunction as a subject recognition unit for calculating at least one ofthe feature quantity of a shape and a feature quantity of motion of thesubject based on the result of the extraction of the subject region bysaid subject extraction unit, and comparing the calculated featurequantity with reference data representing a predetermined featurequantity of the shape or motion, to recognize the shape or the motion ofthe subject.
 14. The operation apparatus as set forth in claim 13,wherein said subject recognition unit calculates the feature quantity ofthe motion based on at least one of the luminance variation and thetexture variation.
 15. The operation apparatus as set forth in claim 13,wherein the one or more processors are further caused to function as: anoperation determination unit for determining a content of the operationbased on the result of the recognition of the shape or motion of thesubject, and generating and outputting a command indicating thedetermined content of the operation.
 16. An imaging method comprising:generating an illumination condition control signal for controlling anillumination condition, and an imaging condition control signal forcontrolling an imaging condition; illuminating a subject with aplurality of mutually different illumination conditions based on theillumination condition control signal; capturing a subject image with animaging condition controlled by the imaging condition control signal togenerate captured images; calculating, using a plurality of capturedimages obtained by capturing images under the different illuminationconditions a luminance variation pertaining to each pixel between theplurality of captured images; calculating, using a plurality of capturedimages obtained by capturing images at different time points, a texturevariation pertaining to each pixel between the plurality of capturedimages; and extracting a subject region based on the luminance variationand the texture variation, wherein calculating said texture variationcalculates, as the texture variation, a first texture variation using aplurality of captured images obtained by capturing images under mutuallyidentical illumination conditions at different time points, and a secondtexture variation using the plurality of captured images which are usedfor the calculation of the luminance variation.
 17. A non-transitorycomputer-readable recording medium having stored thereon a program forcausing a computer to execute the processes of the steps in the methodas set forth in claim 16.